PISA 2012 Creative Problem Solving: International Comparison of High Achievers’ Performance

This post compares the performance of high achievers from selected jurisdictions on the PISA 2012 creative problem solving test.

It draws principally on the material in the OECD Report ‘PISA 2012 Results: Creative Problem Solving’ published on 1 April 2014.

Pisa ps cover CaptureThe sample of jurisdictions includes England, other English-speaking countries (Australia, Canada, Ireland and the USA) and those that typically top the PISA rankings (Finland, Hong Kong, South Korea, Shanghai, Singapore and Taiwan).

With the exception of New Zealand, which did not take part in the problem solving assessment, this is deliberately identical to the sample I selected for a parallel post reviewing comparable results in the PISA 2012 assessments of reading, mathematics and science: ‘PISA 2012: International Comparisons of High Achievers’ Performance’ (December 2013)

These eleven jurisdictions account for nine of the top twelve performers ranked by mean overall performance in the problem solving assessment. (The USA and Ireland lie outside the top twelve, while Japan, Macao and Estonia are the three jurisdictions that are in the top twelve but outside my sample.)

The post is divided into seven sections:

  • Background to the problem solving assessment: How PISA defines problem solving competence; how it defines performance at each of the six levels of proficiency; how it defines high achievement; the nature of the assessment and who undertook it.
  • Average performance, the performance of high achievers and the performance of low achievers (proficiency level 1) on the problem solving assessment. This comparison includes my own sample and all the other jurisdictions that score above the OECD average on the first of these measures.
  • Gender and socio-economic differences amongst high achievers on the problem solving assessment  in my sample of eleven jurisdictions.
  • The relative strengths and weaknesses of jurisdictions in this sample on different aspects of the problem solving assessment. (This treatment is generic rather than specific to high achievers.)
  • What proportion of high achievers on the problem-solving assessment in my sample of jurisdictions are also high achievers in reading, maths and science respectively.
  • What proportion of students in my sample of jurisdictions achieves highly in one or more of the four PISA 2012 assessments – and against the ‘all-rounder’ measure, which is based on high achievement in all of reading, maths and science (but not problem solving).
  • Implications for education policy makers seeking to improve problem solving performance in each of the sample jurisdictions.

Background to the Problem Solving Assessment

.

Definition of problem solving

PISA’s definition of problem-solving competence is:

‘…an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one’s potential as a constructive and reflective citizen.’

The commentary on this definition points out that:

  • Problem solving requires identification of the problem(s) to be solved, planning and applying a solution, and monitoring and evaluating progress.
  • A problem is ‘a situation in which the goal cannot be achieved by merely applying learned procedures’, so the problems encountered must be non-routine for 15 year-olds, although ‘knowledge of general strategies’ may be useful in solving them.
  • Motivational and affective factors are also in play.

The Report is rather coy about the role of creativity in problem solving, and hence the justification for the inclusion of this term in its title.

Perhaps the nearest it gets to an exposition is when commenting on the implications of its findings:

‘In some countries and economies, such as Finland, Shanghai-China and Sweden, students master the skills needed to solve static, analytical problems similar to those that textbooks and exam sheets typically contain as well or better than 15-year-olds, on average, across OECD countries. But the same 15-year-olds are less successful when not all information that is needed to solve the problem is disclosed, and the information provided must be completed by interacting with the problem situation. A specific difficulty with items that require students to be open to novelty, tolerate doubt and uncertainty, and dare to use intuitions (“hunches and feelings”) to initiate a solution suggests that opportunities to develop and exercise these traits, which are related to curiosity, perseverance and creativity, need to be prioritised.’

.

Assessment framework

PISA’s framework for assessing problem solving competence is set out in the following diagram

 

PISA problem solving framework Capture

 

In solving a particular problem it may not be necessary to apply all these steps, or to apply them in this order.

Proficiency levels

The proficiency scale was designed to have a mean score across OECD countries of 500. The six levels of proficiency applied in the assessment each have their own profile.

The lowest, level 1 proficiency is described thus:

‘At Level 1, students can explore a problem scenario only in a limited way, but tend to do so only when they have encountered very similar situations before. Based on their observations of familiar scenarios, these students are able only to partially describe the behaviour of a simple, everyday device. In general, students at Level 1 can solve straightforward problems provided there is a simple condition to be satisfied and there are only one or two steps to be performed to reach the goal. Level 1 students tend not to be able to plan ahead or set sub-goals.’

This level equates to a range of scores from 358 to 423. Across the OECD sample, 91.8% of participants are able to perform tasks at this level.

By comparison, level 5 proficiency is described in this manner:

‘At Level 5, students can systematically explore a complex problem scenario to gain an understanding of how relevant information is structured. When faced with unfamiliar, moderately complex devices, such as vending machines or home appliances, they respond quickly to feedback in order to control the device. In order to reach a solution, Level 5 problem solvers think ahead to find the best strategy that addresses all the given constraints. They can immediately adjust their plans or backtrack when they detect unexpected difficulties or when they make mistakes that take them off course.’

The associated range of scores is from 618 to 683 and 11.4% of all OECD students achieve at this level.

Finally, level 6 proficiency is described in this way:

‘At Level 6, students can develop complete, coherent mental models of diverse problem scenarios, enabling them to solve complex problems efficiently. They can explore a scenario in a highly strategic manner to understand all information pertaining to the problem. The information may be presented in different formats, requiring interpretation and integration of related parts. When confronted with very complex devices, such as home appliances that work in an unusual or unexpected manner, they quickly learn how to control the devices to achieve a goal in an optimal way. Level 6 problem solvers can set up general hypotheses about a system and thoroughly test them. They can follow a premise through to a logical conclusion or recognise when there is not enough information available to reach one. In order to reach a solution, these highly proficient problem solvers can create complex, flexible, multi-step plans that they continually monitor during execution. Where necessary, they modify their strategies, taking all constraints into account, both explicit and implicit.’

The range of level 6 scores is from 683 points upwards and 2.5% of all OECD participants score at this level.

PISA defines high achieving students as those securing proficiency level 5 or higher, so proficiency levels 5 and 6 together. The bulk of the analysis it supplies relates to this cohort, while relatively little attention is paid to the more exclusive group achieving proficiency level 6, even though almost 10% of students in Singapore reach this standard in problem solving.

 .

The sample

Sixty-five jurisdictions took part in PISA 2012, including all 34 OECD countries and 31 partners. But only 44 jurisdictions took part in the problem solving assessment, including 28 OECD countries and 16 partners. As noted above, that included all my original sample of twelve jurisdictions, with the exception of New Zealand.

I could find no stated reason why New Zealand chose not to take part. Press reports initially suggested that England would do likewise, but it was subsequently reported that this decision had been reversed.

The assessment was computer-based and comprised 16 units divided into 42 items. The units were organised into four clusters, each designed to take 20 minutes to complete. Participants completed one or two clusters, depending on whether they were also undertaking computer-based assessments of reading and maths.

In each jurisdiction a random sample of those who took part in the paper-based maths assessment was selected to undertake the problem solving assessment. About 85,000 students took part in all. The unweighted sample sizes in my selected jurisdictions are set out in Table 1 below, together with the total population of 15 year-olds in each jurisdiction.

 

Table 1: Sample sizes undertaking PISA 2012 problem solving assessment in selected jurisdictions

Country Unweighted Sample Total 15 year-olds
Australia 5,612 291,976
Canada 4,601 417,873
Finland 3,531 62,523
Hong Kong 1,325 84,200
Ireland 1,190 59,296
Shanghai 1,203 108,056
Singapore 1,394 53,637
South Korea 1,336 687,104
Taiwan 1,484 328,356
UK (England) 1,458 738,066
USA 1,273 3,985,714

Those taking the assessment were aged between 15 years and three months and 16 years and two months at the time of the assessment. All were enrolled at school and had completed at least six years of formal schooling.

Average performance compared with the performance of high and low achievers

The overall table of mean scores on the problem solving assessment is shown below

PISA problem solving raw scores Capture

 .

There are some familiar names at the top of the table, especially Singapore and South Korea, the two countries that comfortably lead the rankings. Japan is some ten points behind in third place but it in turn has a lead of twelve points over a cluster of four other Asian competitors: Macao, Hong Kong, Shanghai and Taiwan.

A slightly different picture emerges if we compare average performance with the proportion of learners who achieve the bottom proficiency level and the top two proficiency levels. Table 2 below compares these groups.

This table includes all the jurisdictions that exceeded the OECD average score. I have marked out in bold the countries in my sample of eleven which includes Ireland, the only one of them that did not exceed the OECD average.

Table 2: PISA Problem Solving 2012: Comparing Average Performance with Performance at Key Proficiency Levels

 

Jurisdiction Mean score Level 1 (%) Level 5 (%) Level 6 (%) Levels 5+6 (%)
Singapore 562 6.0 19.7 9.6 29.3
South Korea 561 4.8 20.0 7.6 27.6
Japan 552 5.3 16.9 5.3 22.2
Macao 540 6.0 13.8 2.8 16.6
Hong Kong 540 7.1 14.2 5.1 19.3
Shanghai 536 7.5 14.1 4.1 18.2
Taiwan 534 8.2 14.6 3.8 18.4
Canada 526 9.6 12.4 5.1 17.5
Australia 523 10.5 12.3 4.4 16.7
Finland 523 9.9 11.4 3.6 15.0
England (UK) 517 10.8 10.9 3.3 14.2
Estonia 515 11.1 9.5 2.2 11.7
France 511 9.8 9.9 2.1 12.0
Netherlands 511 11.2 10.9 2.7 13.6
Italy 510 11.2 8.9 1.8 10.7
Czech Republic 509 11.9 9.5 2.4 11.9
Germany 509 11.8 10.1 2.7 12.8
USA 508 12.5 8.9 2.7 11.6
Belgium 508 11.6 11.4 3.0 14.4
Austria 506 11.9 9.0 2.0 11.0
Norway 503 13.2 9.7 3.4 13.1
Ireland 498 13.3 7.3 2.1 9.4
OECD Ave. 500 13.2 8.9 2.5 11.4

 .

The jurisdictions at the top of the table also have a familiar profile, with a small ‘tail’ of low performance combined with high levels of performance at the top end.

Nine of the top ten have fewer than 10% of learners at proficiency level 1, though only South Korea pushes below 5%.

Five of the top ten have 5% or more of their learners at proficiency level 6, but only Singapore and South Korea have a higher percentage at level 6 than level 1 (with Japan managing the same percentage at both levels).

The top three performers – Singapore, South Korea and Japan – are the only three jurisdictions that have over 20% of their learners at proficiency levels 5 and 6 together.

South Korea slightly outscores Singapore at level 5 (20.0% against 19.7%). Japan is in third place, followed by Taiwan, Hong Kong and Shanghai.

But at level 6, Singapore has a clear lead, followed by South Korea, Japan, Hong Kong and Canada respectively.

England’s overall place in the table is relatively consistent on each of these measures, but the gaps between England and the top performers vary considerably.

The best have fewer than half England’s proportion of learners at proficiency level 1, almost twice as many learners at proficiency level 5 and more than twice as many at proficiency levels 5 and 6 together. But at proficiency level 6 they have almost three times as many learners as England.

Chart 1 below compares performance on these four measures across my sample of eleven jurisdictions.

All but Ireland are comfortably below the OECD average for the percentage of learners at proficiency level 1. The USA and Ireland are atypical in having a bigger tail (proficiency level 1) than their cadres of high achievers (levels 5 and 6 together).

At level 5 all but Ireland and the USA are above the OECD average, but USA leapfrogs the OECD average at level 6.

There is a fairly strong correlation between the proportions of learners achieving the highest proficiency thresholds and average performance in each jurisdiction. However, Canada stands out by having an atypically high proportion of students at level 6.

.

Chart 1: PISA 2012 Problem-solving: Comparing performance at specified proficiency levels

Problem solving chart 1

.

PISA’s Report discusses the variation in problem-solving performance within different jurisdictions. However it does so without reference to the proficiency levels, so we do not know to what extent these findings apply equally to high achievers.

Amongst those above the OECD average, those with least variation are Macao, Japan, Estonia, Shanghai, Taiwan, Korea, Hong Kong, USA, Finland, Ireland, Austria, Singapore and the Czech Republic respectively.

Perhaps surprisingly, the degree of variation in Finland is identical to that in the USA and Ireland, while Estonia has less variation than many of the Asian jurisdictions. Singapore, while top of the performance table, is only just above the OECD average in terms of variation.

The countries below the OECD average on this measure – listed in order of increasing variation – include England, Australia and Canada, though all three are relatively close to the OECD average. So these three countries and Singapore are all relatively close together.

Gender and socio-economic differences amongst high achievers

 .

Gender differences

On average across OECD jurisdictions, boys score seven points higher than girls on the problem solving assessment. There is also more variation amongst boys than girls.

Across the OECD participants, 3.1% of boys achieved proficiency level 6 but only 1.8% of girls did so. This imbalance was repeated at proficiency level 5, achieved by 10% of boys and 7.7% of girls.

The table and chart below show the variations within my sample of eleven countries. The performance of boys exceeds that of girls in all cases, except in Finland at proficiency level 5, and in that instance the gap in favour of girls is relatively small (0.4%).

 .

Table 3: PISA Problem-solving: Gender variation at top proficiency levels

Jurisdiction Level 5 (%) Level 6 (%) Levels 5+6 (%)
  Boys Girls Diff Boys Girls Diff Boys Girls Diff
Singapore 20.4 19.0 +1.4 12.0 7.1 +4.9 32.4 26.1 +6.3
South Korea 21.5 18.3 +3.2 9.4 5.5 +3.9 30.9 23.8 +7.1
Hong Kong 15.7 12.4 +3.3 6.1 3.9 +2.2 21.8 16.3 +5.5
Shanghai 17.0 11.4 +5.6 5.7 2.6 +3.1 22.7 14.0 +8.7
Taiwan 17.3 12.0 +5.3 5.0 2.5 +2.5 22.3 14.5 +7.8
Canada 13.1 11.8 +1.3 5.9 4.3 +1.6 19.0 16.1 +2.9
Australia 12.6 12.0 +0.6 5.1 3.7 +1.4 17.7 15.7 +2.0
Finland 11.2 11.6 -0.4 4.1 3.0 +1.1 15.3 14.6 +0.7
England (UK) 12.1 9.9 +2.2 3.6 3.0 +0.6 15.7 12.9 +2.8
USA 9.8 7.9 +1.9 3.2 2.3 +0.9 13.0 10.2 +2.8
Ireland 8.0 6.6 +1.4 3.0 1.1 +1.9 11.0 7.7 +3.3
OECD Average 10.0 7.7 +2.3 3.1 1.8 +1.3 13.1 9.5 +3.6

There is no consistent pattern in whether boys are more heavily over-represented at proficiency level 5 than proficiency level 6, or vice versa.

There is a bigger difference at level 6 than at level 5 in Singapore, South Korea, Canada, Australia, Finland and Ireland, but the reverse is true in the five remaining jurisdictions.

At level 5, boys are in the greatest ascendancy in Shanghai and Taiwan while, at level 6, this is true of Singapore and South Korea.

When proficiency levels 5 and 6 are combined, all five of the Asian tigers show a difference in favour of males of 5.5% or higher, significantly in advance of the six ‘Western’ countries in the sample and significantly ahead of the OECD average.

Amongst the six ‘Western’ representatives, boys have the biggest advantage at proficiency level 5 in England, while at level 6 boys in Ireland have the biggest advantage.

Within this group of jurisdictions, the gap between boys and girls at level 6 is comfortably the smallest in England. But, in terms of performance at proficiency levels 5 and 6 together, Finland is ahead.

 .

Chart 2: PISA Problem-solving: Gender variation at top proficiency levels

Problem solving chart 2

The Report includes a generic analysis of gender differences in performance for boys and girls with similar levels of performance in English, maths and science.

It concludes that girls perform significantly above their expected level in both England and Australia (though the difference is only statistically significant in the latter).

The Report comments:

‘It is not clear whether one should expect there to be a gender gap in problem solving. On the one hand, the questions posed in the PISA problem-solving assessment were not grounded in content knowledge, so boys’ or girls’ advantage in having mastered a particular subject area should not have influenced results. On the other hand… performance in problem solving is more closely related to performance in mathematics than to performance in reading. One could therefore expect the gender difference in performance to be closer to that observed in mathematics – a modest advantage for boys, in most countries – than to that observed in reading – a large advantage for girls.’

 .

Socio-economic differences

The Report considers variations in performance against PISA’s Index of Economic, Social and Cultural status (IESC), finding them weaker overall than for reading, maths and science.

It calculates that the overall percentage variation in performance attributable to these factors is about 10.6% (compared with 14.9% in maths, 14.0% in science and 13.2% in reading).

Amongst the eleven jurisdictions in my sample, the weakest correlations were found in Canada (4%), followed by Hong Kong (4.9%), South Korea (5.4%), Finland (6.5%), England (7.8%), Australia (8.5%), Taiwan (9.4%), the USA (10.1%) and Ireland (10.2%) in that order. All those jurisdictions had correlations below the OECD average.

Perhaps surprisingly, there were above average correlations in Shanghai (14.1%) and, to a lesser extent (and less surprisingly) in Singapore (11.1%).

The report suggests that students with parents working in semi-skilled and elementary occupations tend to perform above their expected level in problem-solving in Taiwan, England, Canada, the USA, Finland and Australia (in that order – with Australia closest to the OECD average).

The jurisdictions where these students tend to underperform their expected level are – in order of severity – Ireland, Shanghai, Singapore, Hong Kong and South Korea.

A parallel presentation on the Report provides some additional data about the performance in different countries of what the OECD calls ‘resilient’ students – those in the bottom quartile of the IESC but in the top quartile by perfromance, after accounting for socio-economic status.

It supplies the graph below, which shows all the Asian countries in my sample clustered at the top, but also with significant gaps between them. Canada is the highest-performing of the remainder in my sample, followed by Finland, Australia, England and the USA respectively. Ireland is some way below the OECD average.

.

PISA problem resolving resilience Capture

.

Unfortunately, I can find no analysis of how performance varies according to socio-economic variables at each proficiency level. It would be useful to see which jurisdictions have the smallest ‘excellence gaps’ at levels 5 and 6 respectively.

 .

How different jurisdictions perform on different aspects of problem-solving

The Report’s analysis of comparative strengths and weaknesses in different elements of problem-solving does not take account of variations at different proficiency levels

It explains that aspects of the assessment were found easier by students in different jurisdictions, employing a four-part distinction between:

‘Exploring and understanding. The objective is to build mental representations of each of the pieces of information presented in the problem. This involves:

  • exploring the problem situation: observing it, interacting with it, searching for information and finding limitations or obstacles; and
  • understanding given information and, in interactive problems, information discovered while interacting with the problem situation; and demonstrating understanding of relevant concepts.

Representing and formulating. The objective is to build a coherent mental representation of the problem situation (i.e. a situation model or a problem model). To do this, relevant information must be selected, mentally organised and integrated with relevant prior knowledge. This may involve:

  • representing the problem by constructing tabular, graphic, symbolic or verbal representations, and shifting between representational formats; and
  • formulating hypotheses by identifying the relevant factors in the problem and their inter-relationships; and organising and critically evaluating information.

Planning and executing. The objective is to use one’s knowledge about the problem situation to devise a plan and execute it. Tasks where “planning and executing” is the main cognitive demand do not require any substantial prior understanding or representation of the problem situation, either because the situation is straightforward or because these aspects were previously solved. “Planning and executing” includes:

  • planning, which consists of goal setting, including clarifying the overall goal, and setting subgoals, where necessary; and devising a plan or strategy to reach the goal state, including the steps to be undertaken; and
  • executing, which consists of carrying out a plan.

Monitoring and reflecting.The objective is to regulate the distinct processes involved in problem solving, and to critically evaluate the solution, the information provided with the problem, or the strategy adopted. This includes:

  • monitoring progress towards the goal at each stage, including checking intermediate and final results, detecting unexpected events, and taking remedial action when required; and
  • reflecting on solutions from different perspectives, critically evaluating assumptions and alternative solutions, identifying the need for additional information or clarification and communicating progress in a suitable manner.’

Amongst my sample of eleven jurisdictions:

  • ‘Exploring and understanding’ items were found easier by students in Singapore, Hong Kong, South Korea, Australia, Taiwan and Finland. 
  • ‘Representing and formulating’ items were found easier in Taiwan, Shanghai, South Korea, Singapore, Hong Kong, Canada and Australia. 
  • ‘Planning and executing’ items were found easier in Finland only. 
  • ‘Monitoring and reflecting’ items were found easier in Ireland, Singapore, the USA and England.

The Report concludes:

‘This analysis shows that, in general, what differentiates high-performing systems, and particularly East Asian education systems, such as those in Hong Kong-China, Japan, Korea [South Korea], Macao-China, Shanghai -China, Singapore and Chinese Taipei [Taiwan], from lower-performing ones, is their students’ high level of proficiency on “exploring and understanding” and “representing and formulating” tasks.’

It also distinguishes those jurisdictions that perform best on interactive problems, requiring students to discover some of the information required to solve the problem, rather than being presented with all the necessary information. This seems to be the nearest equivalent to a measure of creativity in problem solving

Comparative strengths and weaknesses in respect of interactive tasks are captured in the following diagram.

.

PISA problem solving strengths in different countries

.

One can see that several of my sample – Ireland, the USA, Canada, Australia, South Korea and Singapore – are placed in the top right-hand quarter of the diagram, indicating stronger than expected performance on both interactive and knowledge acquisition tasks.

England is stronger than expected on the former but not on the latter.

Jurisdictions that are weaker than inspected on interactive tasks only include Hong Kong, Taiwan and Shanghai, while Finland is weaker than expected on both.

We have no information about whether these distinctions were maintained at different proficiency levels.

.

Comparing jurisdictions’ performance at higher proficiency levels

Table 4 and Charts 3 and 4 below show variations in the performance of countries in my sample across the four different assessments at level 6, the highest proficiency level.

The charts in particular emphasise how far ahead the Asian Tigers are in maths at this level, compared with the cross-jurisdictional variation in the other three assessments.

In all five cases, each ‘Asian Tiger’s’ level 6 performance in maths also vastly exceeds its level 6 performance in the other three assessments. The proportion of students achieving level 6 proficiency in problem solving lags far behind, even though there is a fairly strong correlation between these two assessments (see below).

In contrast, all the ‘Western’ jurisdictions in the sample – with the sole exception of Ireland – achieve a higher percentage at proficiency level 6 in problem solving than they do in maths, although the difference is always less than a full percentage point. (Even in Ireland the difference is only 0.1 of a percentage point in favour of maths.)

Shanghai is the only jurisdiction in the sample which has more students achieving proficiency level 6 in science than in problem solving. It also has the narrowest gap between level 6 performance in problem solving and in reading.

Meanwhile, England, the USA, Finland and Australia all have broadly similar profiles across the four assessments, with the largest percentage of level 6 performers in problem solving, followed by maths, science and reading respectively.

The proximity of the lines marking level 6 performance in reading and science is also particularly evident in the second chart below.

.

Table 4: Percentage achieving proficiency Level 6 in each domain

  PS L6  Ma L6 Sci L6 Re L6
Singapore 9.6 19.0 5.8 5.0
South Korea 7.6 12.1 1.1 1.6
Hong Kong 5.1 12.3 1.8 1.9
Shanghai 4.1 30.8 4.2 3.8
Taiwan 3.8 18.0 0.6 1.4
Canada 5.1 4.3 1.8 2.1
Australia 4.4 4.3 2.6 1.9
Finland 3.6 3.5 3.2 2.2
England (UK) 3.3 3.1 1.9 1.3
USA 2.7 2.2 1.1 1.0
Ireland 2.1 2.2 1.5 1.3
OECD Average 2.5 3.3 1.2 1.1

 Charts 3 and 4: Percentage achieving proficiency level 6 in each domain

Problem solving chart 3

Problem solving chart 4

The pattern is materially different at proficiency levels 5 and above, as the table and chart below illustrate. These also include the proportion of all-rounders, who achieved proficiency level 5 or above in each of maths, science and reading (but not in problem-solving).

The lead enjoyed by the ‘Asian Tigers’ in maths is somewhat less pronounced. The gap between performance within these jurisdictions on the different assessments also tends to be less marked, although maths accounts for comfortably the largest proportion of level 5+ performance in all five cases.

Conversely, level 5+ performance on the different assessments is typically much closer in the ‘Western’ countries. Problem solving leads the way in Australia, Canada, England and the USA, but in Finland science is in the ascendant and reading is strongest in Ireland.

Some jurisdictions have a far ‘spikier’ profile than others. Ireland is closest to achieving equilibrium across all four assessments. Australia and England share very similar profiles, though Australia outscores England in each assessment.

The second chart in particular shows how Shanghai’s ‘spike’ applies in all the other three assessments but not in problem solving.

Table 5: Percentage achieving Proficiency level 5 and above in each domain

  PS L5+  Ma L5+ Sci L5+ Re L5+ Ma + Sci + Re L5+
Singapore 29.3 40.0 22.7 21.2 16.4
South Korea 27.6 30.9 11.7 14.2 8.1
Hong Kong 19.3 33.4 16.7 16.8 10.9
Shanghai 18.2 55.4 27.2 25.1 19.6
Taiwan 18.4 37.2 8.4 11.8 6.1
Canada 17.5 16.4 11.3 12.9 6.5
Australia 16.7 14.8 13.5 11.7 7.6
Finland 15.0 15.2 17.1 13.5 7.4
England (UK) 14.2 12.4 11.7 9.1 5.7* all UK
USA 11.6 9.0 7.4 7.9 4.7
Ireland 9.4 10.7 10.8 11.4 5.7
OECD Average 11.4 12.6 8.4 8.4 4.4

 .

Charts 5 and 6: Percentage Achieving Proficiency Level 5 and above in each domain

Problem solving chart 5Problem solving chart 6.

How high-achieving problem solvers perform in other assessments

.

Correlations between performance in different assessments

The Report provides an analysis of the proportion of students achieving proficiency levels 5 and 6 on problem solving who also achieved that outcome on one of the other three assessments: reading, maths and science.

It argues that problem solving is a distinct and separate domain. However:

‘On average, about 68% of the problem-solving score reflects skills that are also measured in one of the three regular assessment domains. The remaining 32% reflects skills that are uniquely captured by the assessment of problem solving. Of the 68% of variation that problem-solving performance shares with other domains, the overwhelming part is shared with all three regular assessment domains (62% of the total variation); about 5% is uniquely shared between problem solving and mathematics only; and about 1% of the variation in problem solving performance hinges on skills that are specifically measured in the assessments of reading or science.’

It discusses the correlation between these different assessments:

‘A key distinction between the PISA 2012 assessment of problem solving and the regular assessments of mathematics, reading and science is that the problem-solving assessment does not measure domain-specific knowledge; rather, it focuses as much as possible on the cognitive processes fundamental to problem solving. However, these processes can also be used and taught in the other subjects assessed. For this reason, problem-solving tasks are also included among the test units for mathematics, reading and science, where their solution requires expert knowledge specific to these domains, in addition to general problem-solving skills.

It is therefore expected that student performance in problem solving is positively correlated with student performance in mathematics, reading and science. This correlation hinges mostly on generic skills, and should thus be about the same magnitude as between any two regular assessment subjects.’

These overall correlations are set out in the table below, which shows that maths has a higher correlation with problem solving than either science or reading, but that this correlation is lower than those between the three subject-related assessments.

The correlation between maths and science (0.90) is comfortably the strongest (despite the relationship between reading and science at the top end of the distribution noted above).

PISA problem solving correlations capture

Correlations are broadly similar across jurisdictions, but the Report notes that the association is comparatively weak in some of these, including Hong Kong. Students here are more likely to perform poorly on problem solving and well on other assessments, or vice versa.

There is also broad consistency at different performance levels, but the Report identifies those jurisdictions where students with the same level of performance exceed expectations in relation to problem-solving performance. These include South Korea, the USA, England, Australia, Singapore and – to a lesser extent – Canada.

Those with lower than expected performance include Shanghai, Ireland, Hong Kong, Taiwan and Finland.

The Report notes:

‘In Shanghai-China, 86% of students perform below the expected level in problem solving, given their performance in mathematics, reading and science. Students in these countries/economies struggle to use all the skills that they demonstrate in the other domains when asked to perform problem-solving tasks.’

However, there is variation according to students’ maths proficiency:

  • Jurisdictions whose high scores on problem solving are mainly attributable to strong performers in maths include Australia, England and the USA. 
  • Jurisdictions whose high scores on problem solving are more attributable to weaker performers in maths include Ireland. 
  • Jurisdictions whose lower scores in problem solving are more attributable to weakness among strong performers in maths include Korea. 
  • Jurisdictions whose lower scores in problem solving are more attributable to weakness among weak performers in maths include Hong Kong and Taiwan. 
  • Jurisdictions whose weakness in problem solving is fairly consistent regardless of performance in maths include Shanghai and Singapore.

The Report adds:

‘In Italy, Japan and Korea, the good performance in problem solving is, to a large extent, due to the fact that lower performing students score beyond expectations in the problem-solving assessment….This may indicate that some of these students perform below their potential in mathematics; it may also indicate, more positively, that students at the bottom of the class who struggle with some subjects in school are remarkably resilient when it comes to confronting real-life challenges in non-curricular contexts…

In contrast, in Australia, England (United Kingdom) and the United States, the best students in mathematics also have excellent problem-solving skills. These countries’ good performance in problem solving is mainly due to strong performers in mathematics. This may suggest that in these countries, high performers in mathematics have access to – and take advantage of – the kinds of learning opportunities that are also useful for improving their problem-solving skills.’

What proportion of high performers in problem solving are also high performers in one of the other assessments?

The percentages of high achieving students (proficiency level 5 and above) in my sample of eleven jurisdictions who perform equally highly in each of the three domain-specific assessments are shown in Table 6 and Chart 7 below.

These show that Shanghai leads the way in each case, with 98.0% of all students who achieve proficiency level 5+ in problem solving also achieving the same outcome in maths. For science and reading the comparable figures are 75.1% and 71.7% respectively.

Taiwan is the nearest competitor in respect of problem solving plus maths, Finland in the case of problem solving plus science and Ireland in the case of problem solving plus reading.

South Korea, Taiwan and Canada are atypical of the rest in recording a higher proportion of problem solving plus reading at this level than problem solving plus science.

Singapore, Shanghai and Ireland are the only three jurisdictions that score above 50% on all three of these combinations. However, the only jurisdictions that exceed the OECD averages in all three cases are Singapore, Hong Kong, Shanghai and Finland.

Table 6: PISA problem-solving: Percentage of students achieving proficiency level 5+ in domain-specific assessments

  PS + Ma PS + Sci PS + Re
Singapore 84.1 57.0 50.2
South Korea 73.5 34.1 40.3
Hong Kong 79.8 49.4 48.9
Shanghai 98.0 75.1 71.7
Taiwan 93.0 35.3 43.7
Canada 57.7 43.9 44.5
Australia 61.3 54.9 47.1
Finland 66.1 65.4 49.5
England (UK) 59.0 52.8 41.7
USA 54.6 46.9 45.1
Ireland 59.0 57.2 52.0
OECD Average 63.5 45.7 41.0

Chart 7: PISA Problem-solving: Percentage of students achieving proficiency level 5+ in domain-specific assessments

Problem solving chart 7.

What proportion of students achieve highly in one or more assessments?

Table 7 and Chart 8 below show how many students in each of my sample achieved proficiency level 5 or higher in problem-solving only, in problem solving and one or more assessments, in one or more assessments but not problem solving and in at least one assessment (ie the total of the three preceding columns).

I have also repeated in the final column the percentage achieving this proficiency level in each of maths, science and reading. (PISA has not released information about the proportion of students who achieved this feat across all four assessments.)

These reveal that the percentages of students who achieve proficiency level 5+ only in problem solving are very small, ranging from 0.3% in Shanghai to 6.7% in South Korea.

Conversely, the percentages of students achieving proficiency level 5+ in any one of the other assessments but not in problem solving are typically significantly higher, ranging from 4.5% in the USA to 38.1% in Shanghai.

There is quite a bit of variation in terms of whether jurisdictions score more highly on ‘problem solving and at least one other’ (second column) and ‘at least one other excluding problem solving (third column).

More importantly, the fourth column shows that the jurisdiction with the most students achieving proficiency level 5 or higher in at least one assessment is clearly Shanghai, followed by Singapore, Hong Kong, South Korea and Taiwan in that order.

The proportion of students achieving this outcome in Shanghai is close to three times the OECD average, comfortably more than twice the rate achieved in any of the ‘Western’ countries and three times the rate achieved in the USA.

The same is true of the proportion of students achieving this level in the three domain-specific assessments.

On this measure, South Korea and Taiwan fall significantly behind their Asian competitors, and the latter is overtaken by Australia, Finland and Canada.

 .

Table 7: Percentage achieving proficiency level 5+ in different combinations of PISA assessments

  PS only% PS + 1 or more% 1+ butNot PS% L5+ in at least one % L5+ in Ma + Sci + Re %
Singapore 4.3 25.0 16.5 45.8 16.4
South Korea 6.7 20.9 11.3 38.9 8.1
Hong Kong 3.4 15.9 20.5 39.8 10.9
Shanghai 0.3 17.9 38.1 56.3 19.6
Taiwan 1.2 17.1 20.4 38.7 6.1
Canada 5.5 12.0 9.9 27.4 6.5
Australia 4.7 12.0 7.7 24.4 7.6
Finland 3.0 12.0 11.9 26.9 7.4
England (UK) 4.4 9.8 6.8 21.0 5.7* all UK
USA 4.1 7.5 4.5 16.1 4.7
Ireland 2.6 6.8 10.1 19.5 5.7
OECD Average 3.1 8.2 8.5 19.8 4.4

Chart 8: Percentage achieving proficiency level 5+ in different combinations of PISA assessments

Problem solving chart 8

The Report comments:

The proportion of students who reach the highest levels of proficiency in at least one domain (problem solving, mathematics, reading or science) can be considered a measure of the breadth of a country’s/economy’s pool of top performers. By this measure, the largest pool of top performers is found in Shanghai-China, where more than half of all students (56%) perform at the highest levels in at least one domain, followed by Singapore (46%), Hong  Kong-China (40%), Korea and Chinese  Taipei (39%)…Only one OECD country, Korea, is found among the five countries/economies with the largest proportion of top performers. On average across OECD countries, 20% of students are top performers in at least one assessment domain.

The proportion of students performing at the top in problem solving and in either mathematics, reading or science, too can be considered a measure of the depth of this pool. These are top performers who combine the mastery of a specific domain of knowledge with the ability to apply their unique skills flexibly, in a variety of contexts. By this measure, the deepest pools of top performers can be found in Singapore (25% of students), Korea (21%), Shanghai-China (18%) and Chinese Taipei (17%). On average across OECD countries, only 8% of students are top performers in both a core subject and in problem solving.’

There is no explanation of why proficiency level 5 should be equated by PISA with the breadth of a jurisdiction’s ‘pool of top performers’. The distinction between proficiency levels 5 and 6 in this respect requires further discussion.

In addition to updated ‘all-rounder’ data showing what proportion of students achieved this outcome across all four assessments, it would be really interesting to see the proportion of students achieving at proficiency level 6 across different combinations of these four assessments – and to see what proportion of students achieving that outcome in different jurisdictions are direct beneficiaries of targeted support, such as a gifted education programme.

In the light of this analysis, what are jurisdictions’ priorities for improving  problem solving performance?

Leaving aside strengths and weaknesses in different elements of problem solving discussed above, this analysis suggests that the eleven jurisdictions in my sample should address the following priorities:

Singapore has a clear lead at proficiency level 6, but falls behind South Korea at level 5 (though Singapore re-establishes its ascendancy when levels 5 and 6 are considered together). It also has more level 1 performers than South Korea. It should perhaps focus on reducing the size of this tail and pushing through more of its mid-range performers to level 5. There is a pronounced imbalance in favour of boys at level 6, so enabling more girls to achieve the highest level of performance is a clear priority. There may also be a case for prioritising the children of semi-skilled workers.

South Korea needs to focus on getting a larger proportion of its level 5 performers to level 6. This effort should be focused disproportionately on girls, who are significantly under-represented at both levels 5 and 6. South Korea has a very small tail to worry about – and may even be getting close to minimising this. It needs to concentrate on improving the problem solving skills of its stronger performers in maths.

Hong Kong has a slightly bigger tail than Singapore’s but is significantly behind at both proficiency levels 5 and 6. In the case of level 6 it is equalled by Canada. Hong Kong needs to focus simultaneously on reducing the tail and lifting performance across the top end, where girls and weaker performers in maths are a clear priority.

Shanghai has a similar profile to Hong Kong’s in all respects, though with somewhat fewer level 6 performers. It also needs to focus effort simultaneously at the top and the bottom of the distribution. Amongst this sample, Shanghai has the worst under-representation of girls at level 5 and levels 5 and 6 together, so addressing that imbalance is an obvious priority. It also demonstrated the largest variation in performance against PISA’s IESC index, which suggests that it should target young people from disadvantaged backgrounds, as well as the children of semi-skilled workers.

Taiwan is rather similar to Hong Kong and Shanghai, but its tail is slightly bigger and its level 6 cadre slightly smaller, while it does somewhat better at level 5. It may need to focus more at the very bottom, but also at the very top. Taiwan also has a problem with high-performing girls, second only to Shanghai as far as level 5 and levels 5 and 6 together are concerned. However, like Shanghai, it does comparatively better than the other ‘Asian Tigers’ in terms of girls at level 6. It also needs to consider the problem solving performance of its weaker performers in maths.

Canada is the closest western competitor to the ‘Asian Tigers’ in terms of the proportions of students at levels 1 and 5 – and it already outscores Shanghai and Taiwan at level 6. It needs to continue cutting down the tail without compromising achievement at the top end. Canada also has small but significant gender imbalances in favour of boys at the top end.

Australia by comparison is significantly worse than Canada at level 1, broadly comparable at level 5 and somewhat worse at level 6. It too needs to improve scores at the very bottom and the very top. Australia’s gender imbalance is more pronounced at level 6 than level 5.

Finland has the same mean score as Australia’s but a smaller tail (though not quite as small as Canada’s). It needs to improve across the piece but might benefit from concentrating rather more heavily at the top end. Finland has a slight gender imbalance in favour of girls at level 5, but boys are more in the ascendancy at level 6 than in either England or the USA. As in Australia, this latter point needs addressing.

England has a profile similar to Australia’s, but less effective at all three selected proficiency levels. It is further behind at the top than at the bottom of the distribution, but needs to work hard at both ends to catch up the strongest western performers and maintain its advantage over the USA and Ireland. Gender imbalances are small but nonetheless significant.

USA has a comparatively long tail of low achievement at proficiency level 1 and, with the exception of Ireland, the fewest high achievers. This profile is very close to the OECD average. As in England, the relatively small size of gender imbalances in favour of boys does not mean that these can be ignored.

Ireland has the longest tail of low achievement and the smallest proportion of students at proficiency levels 5, 6 and 5 and 6 combined. It needs to raise the bar at both ends of the achievement distribution. Ireland has a larger preponderance of boys at level 6 than its Western competitors and this needs addressing. The limited socio-economic evidence suggests that Ireland should also be targeting the offspring of parents with semi-skilled and elementary occupations.

So there is further scope for improvement in all eleven jurisdictions. Meanwhile the OECD could usefully provide a more in-depth analysis of high achievers on its assessments that features:

  • Proficiency level 6 performance across the board.
  • Socio-economic disparities in performance at proficiency levels 5 and 6.
  • ‘All-rounder’ achievement at these levels across all four assessments and
  • Correlations between success at these levels and specific educational provision for high achievers including gifted education programmes.

.

GP

April 2014

Unpacking the Primary Assessment and Accountability Reforms

This post examines the Government response to consultation on primary assessment and accountability.

pencil-145970_640It sets out exactly what is planned, what further steps will be necessary to make these plans viable and the implementation timetable.

It is part of a sequence of posts I have devoted to this topic, most recently:

Earlier posts in the series include The Removal of National Curriculum Levels and the Implications for Able Pupils’ Progression (June 2012) and Whither National Curriculum Assessment Without Levels? (February 2013).

The consultation response contrives to be both minimal and dense. It is necessary to unpick each element carefully, to consider its implications for the package as a whole and to reflect on how that package fits in the context of wider education reform.

I have organised the post so that it considers sequentially:

  • The case for change, including the aims and core principles, to establish the policy frame for the planned reforms.
  • The impact on the assessment experience of children aged 2-11 and how that is likely to change.
  • The introduction of baseline assessment in Year R.
  • The future shape of end of KS1 and end of KS2 assessment respectively.
  • How the new assessment outcomes will be derived, reported and published.
  • The impact on floor standards.

Towards the end of the post I have also provided a composite ‘to do’ list containing all the declared further steps necessary to make the plan viable, with a suggested deadline for each.

And the post concludes with an overall judgement on the plans, in the form of a summary of key issues and unanswered questions arising from the earlier commentary. Impatient readers may wish to jump straight to that section.

I am indebted to Warwick Mansell for his previous post on this topic. I shall try hard not to parrot the important points he has already made, though there is inevitably some overlap.

Readers should also look to Michael Tidd for more information about the shape and content of the new tests.

What has been published?

The original consultation document ‘Primary assessment and accountability under the new national curriculum’ was published on 17 July 2013 with a deadline for response of 17 October 2013. At that stage the Government’s response was due ‘in autumn 2013’.

The response was finally published on 27 March, some four months later than planned and only five months prior to the introduction of the revised national curriculum which these arrangements are designed to support.

It is likely that the Government will have decided that 31 March was the latest feasible date to issue the response, so they were right up against the wire.

It was accompanied by:

  • A press release which focused on the full range of assessment reforms – for primary, secondary and post-16.

Shortly before the response was published, the reply to a Parliamentary question asked on 17 March explained that test frameworks were expected to be included within it:

‘Guidance on the nature of the revised key stage 1 and key stage 2 tests, including mathematics, will be published by the Standards and Testing Agency in the form of test framework documents. The frameworks are due to be released as part of the Government’s response to the primary assessment and accountability consultation. In addition, some example test questions will be made available to schools this summer and a full sample test will be made available in the summer of 2015.’ (Col 383W)

.

.

In the event, these documents – seven in all – did not appear until 31 March and there was no reference to any of the three commitments above in what appeared on 27 March.

Finally, the Standards and Testing Agency published on 3 April a guidance page on national curriculum tests from 2016. At present it contains very little information but further material will be added as and when it is published.

Partly because the initial consultation document was extremely ‘drafty’, the reaction of many key external respondents to the consultation was largely negative. One imagines that much of the period since 17 October has been devoted to finding the common ground.

Policy makers will have had to do most of their work after the consultation document issued because they were not ready beforehand.

But the length of the delay in issuing the response would suggest that they also encountered significant dissent amongst internal stakeholders – and that the eventual outcome is likely to be a compromise of sorts between these competing interests.

Such compromises tend to have observable weaknesses and/or put off problematic issues for another day.

A brief summary of consultation responses is included within the Government’s response. I will refer to this at relevant points during the discussion below.

 .

The Case for Change

 .

Aims

The consultation response begins – as did the original consultation document – with a section setting out the case for reform.

It provides a framework of aims and principles intended to underpin the changes that are being set in place.

The aims are:

  • The most important outcome of primary education is to ‘give as many pupils as possible the knowledge and skills to flourish in the later phases of education’. This is a broader restatement of the ‘secondary ready’ concept adopted in the original consultation document.
  • The primary national curriculum and accountability reforms ‘set high expectations so that all children can reach their potential and are well prepared for secondary school’. Here the ‘secondary ready’ hurdle is more baldly stated. The parallel notion is that all children should do as well as they can – and that they may well achieve different levels of performance. (‘Reach their potential’ is disliked by some because it is considered to imply a fixed ceiling for each child and fixed mindset thinking.)
  • To raise current threshold expectations. These are set too low, since too few learners (47%) with KS2 level 4C in both English and maths go on to achieve five or more GCSE grades A*-C including English and maths, while 72% of those with KS2 level 4B do so. So the new KS2 bar will be set at this higher level, but with the expectation that 85% of learners per school will jump it, 13% more than the current national figure. Meanwhile the KS4 outcome will also change, to achievement across eight GCSEs rather than five, quite probably at a more demanding level than the present C grade. In the true sense, this is a moving target.
  • No child should be allowed to fall behind’. This is a reference to the notion of ‘mastery’ in its crudest sense, though the model proposed will not deliver this outcome. We have noted already a reference to ‘as many children as possible’ and the school-level target – initially at least – will be set at 85%. In reality, a significant minority of learners will progress more slowly and will fall short of the threshold at the end of KS2.
  • The new system ‘will set a higher bar’ but ‘almost all pupils should leave primary school well-placed to succeed in the next phase of their education’. Another nuanced version of ‘secondary ready’ is introduced. This marks a recognition that some learners will not jump over the higher bar. In the light of subsequent references to 85%, ‘almost all’ is rather over-optimistic.
  • We also want to celebrate the progress that pupils make in schools with more challenging intakes’. Getting ‘nearly all pupils to meet this standard…’ (the standard of secondary readiness?) ‘…is very demanding, at least in the short term’. There will therefore be recognition of progress ‘from a low starting point’ – even though these learners have, by definition, been allowed to fall behind and will continue to do so.

So there is something of a muddle here, no doubt engendered by a spirit of compromise.

The black and white distinction of ‘secondary-readiness’ has been replaced by various verbal approximations, but the bottom line is that there will be a defined threshold denoting preparedness that is pitched higher than the current threshold.

And the proportion likely to fall short is downplayed – there is apparent unwillingness at this stage to acknowledge the norm that up to 15% of learners in each school will undershoot the threshold – substantially more in schools with ‘challenging intakes’.

What this boils down to is a desire that all will achieve the new higher hurdle – and that all will be encouraged to exceed it if they can – tempered by recognition that this is presently impossible. No child should be allowed to fall behind but many inevitably will do so.

It might have been better to express these aims in the form of future aspirations – and our collective efforts to bridge the gap between present reality and those ambitious aspirations.

Principles

The section concludes with a new set of principles governing pedagogy, assessment and accountability:

  • ‘Ongoing, teacher-led assessment is a crucial part of effective teaching;
  • Schools should have the freedom to decide how to teach their curriculum and how to track the progress that pupils make;
  • Both summative teacher assessment and external testing are important;
  • Accountability is key to a successful school system, and therefore must be fair and transparent;
  • Measures of both progress and attainment are important for understanding school performance; and
  • A broad range of information should be published to help parents and the wider public know how well schools are performing.’

These are generic ‘motherhood and apple pie’ statements and so largely uncontroversial. I might have added a seventh – that schools’ in-house assessment and reporting systems must complement summative assessment and testing, including by predicting for parents the anticipated outcomes of the latter.

Perhaps interestingly, there is no repetition of the defence for the removal of national curriculum levels. Instead, the response concentrates on the support available to schools.

It mentions discussion with an ‘expert group on assessment’ about ‘how to support schools to make best use of the new assessment freedoms’. We are not told the membership of this group (which, as far as I know, has not been made public) or the nature of its remit.

There is also a link to information about the Assessment Innovation Fund, which will provide up to 10 grants of up to £10,000 which schools and organisations can use to develop packages that share their innovative practice with others.

 

Children’s experience of assessment up to the end of KS2

The response mentions the full range of national assessments that will impact on children between the ages of two and 11:

  • The statutory progress check at two years of age.
  • A new baseline assessment undertaken within a few weeks of the start of Year R, introduced from September 2015.
  • An Early Years Foundation Stage Profile undertaken in the final term of the year in which children reach the age of five. A revised profile was introduced from September 2012. It is currently compulsory but will be optional from September 2016. The original consultation document said that the profile would no longer be moderated and data would no longer be collected. Neither of those commitments is repeated here.
  • The Phonics Screening Check, normally undertaken in Year 1. The possibility of making these assessments non-statutory for all-through primary schools, suggested in the consultation document, has not been pursued: 53% of respondents opposed this idea, whereas 32% supported it.
  • End of KS1 assessment and
  • End of KS2 assessment.

So a total of six assessments are in place between the ages of two and 11. At least four – and possibly five – will be undertaken between ages two and seven.

It is likely that early years’ professionals will baulk at this amount of assessment, no matter how sensitively it is designed. But the cost and inefficiency of the model is also open to criticism.

The Reception Baseline

Approach

The original consultation document asked whether:

  • KS1 assessment should be retained as a baseline – 45% supported this and 41% were opposed.
  • A baseline check should be introduced at the start of Reception – 51% supported this and 34% were opposed.
  • Such a baseline check should be optional – 68% agreed and 19% disagreed.
  • Schools should be allowed to choose from a range of commercially available materials for this baseline check – 73% said no and only 15% said yes.

So, whereas views were mixed on where the baseline should be set, there were substantial majorities in favour of any Year R baseline check being optional and following a single, standard national format.

The response argues that Year R is the most sensible point at which to position the baseline since that is:

‘…the earliest point that nearly all children are in school’.

What happens in respect of children who are not in school at this point is not discussed.

There is no explanation of why the Government has disregarded the clear majority of respondents by choosing to permit a range of assessment approaches, so this decision must be ideologically motivated.

The response says ‘most’ are likely to be administered by teaching staff, leaving open the possibility that some options will be administered externally.

Design

Such assessments will need to be:

‘…strong predictors of key stage 1 and key stage 2 attainment, whilst reflecting the age and abilities of children in Reception’.

Presumably this means predictors of attainment in each of the three core subjects – English, maths and science – rather than any broader notion of attainment. The challenge inherent in securing a reasonable predictor of attainment across these domains seven years further on in a child’s development should not be under-estimated.

The response points out that such assessment tools are already available for use in Year R, some are used widely and some schools have long experience of using them. But there is no information about how many of these are deemed to meet already the description above.

In any case, new criteria need to be devised which all such assessments must meet. Some degree of modification will be necessary for all existing products and new products will be launched to compete in the market.

There is an opportunity to use this process to ratchet up the Year R Baseline beyond current expectations, so matching the corresponding process at the end of KS2. The consultation response says nothing about whether this is on the cards.

Interestingly, in his subsequent ‘Unsure start’ speech about early years inspection, HMCI refers to:

‘…the government’s announcement last week that they will be introducing a readiness-for-school test at age four. This is an ideal opportunity to improve accountability. But I think it should go further.

I hope that the published outcomes of these tests will be detailed enough to show parents how their own child has performed. I fear that an overall school grade will fail to illuminate the progress of poor children. I ask government to think again about this issue.’

The terminology – ‘readiness for school’ is markedly blunter than the references to a reception baseline in the consultation response. There is nothing in the response about the outcomes of these tests being published, nor anything about ‘an overall school grade’.

Does this suggest that decisions have already been made that were not communicated in the consultation response?

.

Timeline, options, questions

Several pieces of further work are required in short order to inform schools and providers about what will be required – and to enable both to prepare for introduction of the assessments from September 2015. All these should feature in the ‘to do’ list below.

One might reasonably have hoped that – especially given the long delay – some attempt might have been made to publish suggested draft criteria for the baseline alongside the consultation response. The fact that even preliminary research into existing practice has not been undertaken is a cause for concern.

Although the baseline will be introduced from September 2015, there is a one-year interim measure which can only apply to all-through primary schools:

  • They can opt out of the Year R baseline measure entirely, relying instead on KS1 outcomes as their baseline; or
  • They can use an approved Year R baseline assessment and have this cohort’s progress measured at the end of KS2 (which will be in 2022) by either the Year R or the KS1 baseline, whichever demonstrates the most progress.

In the period up to and including 2021, progress will continue to be measured from the end of KS1. So learners who complete KS2 in 2021 for example will be assessed on progress since their KS1 tests in 2017.

Junior and middle schools will also continue to use a KS1 baseline.

Arrangements for infant and first schools are still to be determined, another rather worrying omission at this stage in proceedings.

It is also clear that all-through primary schools (and infant/first schools?) will continue to be able to opt out from the Year R baseline from September 2016 onwards, since the response says:

‘Schools that choose not to use an approved baseline assessment from 2016 will be judged on an attainment floor standard alone’.

Hence the Year R baseline check is entirely optional and a majority of schools could choose not to undertake it.

However, they would need to be confident of meeting the demanding 85% attainment threshold in the floor standard.

They might be wise to postpone that decision until the pitch of the progress expectation is determined. For neither the Year R baseline nor the amount of progress that learners are expected to make from their starting point in Year R is yet defined.

This latter point applies at the average school level (for the purposes of the floor standard) and in respect of the individual learner. For example, if a four year-old is particularly precocious in, say, maths, what scaled scores must they register seven years later to be judged to have made sufficient progress?

There are several associated questions that follow on from this.

Will it be in schools’ interests to acknowledge that they have precocious four year-olds at all? Will the Year R baseline reinforce the tendency to use Reception to bring all children to the same starting point in readiness for Year 1, regardless of their precocity?

Will the moderation arrangements be hard-edged enough to stop all-through primary schools gaming the system by artificially depressing their baseline outcomes?

Who will undertake this moderation and how much will it cost? Will not the decision to permit schools to choose from a range of measures unnecessarily complicate the moderation process and add to the expense?

The consultation response neither poses these questions nor supplies answers.

The future shape of end KS1 and end KS2 assessment

.

What assessment will take place?

At KS1 learners will be assessed in:

  • Reading – test plus teacher assessment
  • Writing – test (of grammar, punctuation and spelling) plus teacher assessment
  • Speaking and listening – teacher assessment
  • Maths – test plus teacher assessment
  • Science  - teacher assessment

The new test of grammar, punctuation and spelling did not feature in the original consultation and has presumably been introduced to strengthen the marker of progress to which four year-olds should aspire at age seven.

The draft test specifications for the KS1 tests in reading, GPS and maths outline the requirements placed on the test developers, so it is straightforward to compare the specifications for reading and maths with the current tests.

The GPS test will include a 20 minute written grammar and punctuation task; a 20 minute test comprising short grammar, punctuation and vocabulary questions; and a 15 minute spelling task.

There is a passing reference to further work on KS1 moderation which is included in the ‘to do’ list below.

At KS2 learners will be assessed in

  • Reading – test plus teacher assessment
  • Writing – test (of grammar spelling and punctuation) plus teacher assessment
  • Maths – test plus teacher assessment
  • Science  - teacher assessment plus a science sampling test.

Once again, the draft test specifications – reading, GPS, maths and science sampling – describe the shape of each test and the content they are expected to assess.

I will leave it to experts to comment on the content of the tests.

 .

Academies and free schools

It is important to note that the framing of this content – by means of detailed ‘performance descriptors’ – means that the freedom academies and free schools enjoy in departing from the national curriculum will be largely illusory.

I raised this issue back in February 2013:

  • ‘We know that there will be a new grading system in the core subjects at the end of KS2. If this were to be based on the ATs as drafted, it could only reflect whether or not learners can demonstrate that they know, can apply and understand ‘the matters, skills and processes specified’ in the PoS as a whole. Since there is no provision for ATs that reflect sub-elements of the PoS – such as reading, writing, spelling – grades will have to be awarded on the basis of separate syllabuses for end of KS2 tests associated with these sub-elements.
  • This grading system must anyway be applied universally if it is to inform the publication of performance tables. Since some schools are exempt from National Curriculum requirements, it follows that grading cannot be derived directly from the ATs and/or the PoS, but must be independent of them. So this once more points to end of KS2 tests based on entirely separate syllabuses which nevertheless reflect the relevant part of the draft PoS. The KS2 arrangements are therefore very similar to those planned at KS4.’

I have more to say about the ‘performance descriptors’ below.

 .

Single tests for all learners

A critical point I want to emphasise at this juncture – not mentioned at all in the consultation document or the response – is the test development challenge inherent in producing single papers suitable for all learners, regardless of their attainment.

We know from the response that the P-scales will be retained for those who are unable to access the end of key stage tests. (Incidentally, the content of the P-scales will remain unchanged so they will not be aligned with the revised national curriculum, as suggested in the consultation document.)

There will also be provision for pupils who are working ‘above the P-scales but below the level of the test’.

Now the P-scales are for learners working below level 1 (in old currency). This is the first indication I have seen that the tests may not cater for the full range from Level 1-equivalent to Level 6-equivalent and above. But no further information is provided.

It may be that this is a reference to learners who are working towards level 1 (in old currency) but do not have SEN.

The 2014 KS2 ARA booklet notes:

‘Children working towards level 1 of the national curriculum who do not have a special educational need should be reported to STA as ‘W’ (Working below the level). This includes children who are working towards level 1 solely because they have English as an additional language. Schools should use the code ‘NOTSEN’ to explain why a child working towards level 1 does not have P scales reported. ‘NOTSEN’ replaces the code ‘EAL’ that was used in previous years.’

The consultation document said:

‘We do not propose to develop an equivalent to the current level 6 tests, which are used to challenge the highest-attaining pupils. Key stage 2 national curriculum tests will include challenging material (at least of the standard of the current level 6 test) which all pupils will have the opportunity to answer, without the need for a separate test’.

The draft test specifications make it clear that the tests should:

‘provide a suitable challenge for all children and give every child the opportunity to achieve as high a standard…as possible.’

Moreover:

‘In order to improve general accessibility for all children, where possible, questions will be placed in order of difficulty.’

The development of single tests covering this span of attainment – from level 1 to above level 6 – tests in which the questions are posed in order of difficulty and even the highest attainers must answer all questions – seem to me to be a very tall order, especially in maths.

More than that, I urgently need persuading that this is not a waste of high attainers’ time and poor assessment practice.

 .

How assessment outcomes will be derived, reported and published

Deriving assessment outcomes

One of the reasons cited for replacing national curriculum levels was the complexity of the system and the difficulty parents experienced in understanding it.

The Ministerial response to the original report from the National Curriculum Expert Panel said:

‘As you rightly identified, the current system is confusing for parents and restrictive for teachers. I agree with your recommendation that there should be a direct relationship between what children are taught and what is assessed. We will therefore describe subject content in a way which makes clear both what should be taught and what pupils should know and be able to do as a result.’

The consultation document glossed the same point thus:

‘Schools will be able to focus their teaching, assessment and reporting not on a set of opaque level descriptions, but on the essential knowledge that all pupils should learn.’

However, the consultation response introduces for the first time the concept of a ‘performance descriptor’.

This term is defined in the glossaries at the end of each draft test specification:

Description of the typical characteristics of children working at a particular standard. For these tests, the performance descriptor will characterise the minimum performance required to be working at the appropriate standard for the end of the key stage.’

Essentially this is a collective term for something very similar to old-style level descriptions.

Except that, in the case of the tests, they are all describing the same level of performance.

They have been rendered necessary by the odd decision to provide only a single generic attainment target for each programme of study. But, as noted back in February 2013, the test developers need a more sophisticated framework on which to base their assessments.

According to the draft test specifications they will also be used

‘By a panel of teachers to set the standards on the new tests following their first administration in May 2016’.

When it comes to teacher assessment, the consultation response says:

‘New performance descriptors will be introduced to inform the statutory teacher assessments at the end of key stage one [and]…key stage two.’

But there are two models in play simultaneously.

In four cases – science at KS1 and reading, maths and science at KS2 – there will be ‘a single performance descriptor of the new expected standard’, in the same way as there are in the test specifications.

But in five cases – reading, writing, speaking and listening and maths at KS1; and writing at KS2 :

‘teachers will assess pupils as meeting one of several performance descriptors’.

These are old-style level descriptors by another name. They perform exactly the same function.

The response says that the KS1 teacher assessment performance descriptors will be drafted by an expert group for introduction in autumn 2014. It does not mention whether KS2 teacher assessment performance descriptors will be devised in the same way and to the same timetable.

 .

Reporting assessment outcomes to parents

When it comes to reporting to parents, there will be three different arrangements in play at both KS1 and KS2:

  • Test results will be reported by means of scaled scores (of which more in a moment).
  • One set of teacher assessments will be reported by selecting from a set of differentiated performance descriptors.
  • A second set of teacher assessments will be reported according to whether learners have achieved a single threshold performance descriptor.

This is already significantly more complex than the previous system, which applied the same framework of national curriculum levels across the piece.

It seems that KS1 test outcomes will be reported as straightforward scaled scores (though this is only mentioned on page 8 of the main text of the response and not in Annex B, which compares the new arrangements with those currently in place).

But, in the case of KS2:

‘Parents will be provided with their child’s score alongside the average for their school, the local area and nationally. In the light of the consultation responses, we will not give parents a decile ranking for their child due to concerns about whether decile rankings are meaningful and their reliability at individual pupil level.’

The consultation document proposed a tripartite reporting system comprising:

  • A scaled score for each KS2 test, derived from raw test marks and built around a ‘secondary readiness standard’. This standard would be set at a scaled score of 100, which would remain unchanged. It was suggested for illustrative purposes that a scale based on the current national curriculum tests might run from 80 to 130.
  • An average scaled score in each test for other pupils nationally with the same prior attainment at the baseline. Comparison of a learner’s scaled score with the average scaled score would show whether they had made more or less progress than the national average.
  • A national ranking in each test – expressed in terms of deciles – showing how a learner’s scaled score compared with the range of performance nationally.

The latter has been dispensed with, given that 35% of consultation respondents disagreed with it, but there were clearly technical reservations too.

In its place, the ‘value added’ progress measure has been expanded so that there is a comparison with other pupils in the learner’s own school and the ‘local area’ (which presumably means local authority). This beefs up the progression element in reporting at the expense of information about the attainment level achieved.

So at the end of KS2 parents will receive scaled scores and three average scaled scores for each of reading, writing and maths – twelve scores in all – plus four performance descriptors, of which three will be singleton threshold descriptors (reading, maths and science) and one will be selected from a differentiated series (writing). That makes sixteen assessment outcomes altogether, provided in four different formats.

The consultation response tells us nothing more about the range of the scale that will be used to provide scaled scores. We do not even know if it will be the same for each test.

The draft test specifications say that:

‘The exact scale for the scaled scores will be determined following further analysis of trialling data. This will include a full review of the reporting of confidence intervals for scaled scores.’

But they also contain this worrying statement:

‘The provision of a scaled score will aid in the interpretation of children’s performance over time as the scaled score which represents the expected standard will be the same year on year. However, at the extremes of the scaled score distribution, as is standard practice, the scores will be truncated such that above and below a certain point, all children will be awarded the same scaled score in order to minimise the effect for children at the ends of the distribution where the test is not measuring optimally.’

This appears to suggest that scaled scores will not accurately describe performance at the extremes of the distribution, because the tests will not accurately measure such performance. This might be describing a statistical truism, but it again begs the question whether the highest attainers are being short-changed by the selected approach.

.

Publication of assessment outcomes

The response introduces the idea that ‘a suite of indicators’ will be published on each school’s own website in a standard format. These are:

  • The average progress made by pupils in reading, writing and maths. (This is presumably relevant to both KS1 and KS2 and to both tests and teacher assessment.)
  • The percentage of pupils reaching the expected standard in reading, writing and mathematics at the end of key stage 2. (This is presumably relevant to both tests and teacher assessment.)
  • The average score of pupils in their end of key stage 2 assessments. (The final word suggests teacher assessment as well as tests, even though there will not be a score from the former.)
  • The percentage of pupils who achieve a high score in all areas at the end of key stage 2. (Does ‘all areas’ imply something more than statutory tests and teacher assessments? Does it mean treating each area separately, or providing details only of those who have achieved high scores across all areas?)

The latter is the only reference to high attainers in the entire response. It does not give any indication of what will count as a high score for these purposes. Will it be designed to catch the top-third of attainers or something more demanding, perhaps equivalent to the top decile?

A decision has been taken not to report the outcomes of assessment against the P-scales because the need to contextualise such information is perceived to be relatively greater.

And, as noted above, HMCI let slip the fact that the outcomes of reception baselines would also be published, but apparently in the form of a single overall grade.

We are not told when these requirements will be introduced, but presumably they must be in place to report the outcomes of assessments undertaken in spring 2016.

Additionally:

‘So that parents can make comparisons between schools, we would like to show each school’s position in the country on these measures and present these results in a manner that is clear for all audiences to understand. We will discuss how best to do so with stakeholders, to ensure that the presentation of the data is clear, fair and statistically robust.’

This suggests inclusion in the 2016 School Performance Tables, but this is not stated explicitly.

Indeed, apart from references to the publication of progress measures in the 2022 Performance Tables, there is no explicit coverage of their contribution in the response, nor any reference to the planned supporting data portal, or how data will be distributed between the Tables and the portal.

The original consultation document gave several commitments on the future content of performance tables. They included:

  • How many of a school’s pupils are amongst the highest attaining nationally, by showing the percentage of pupils achieving a high scaled score in each subject.
  • Measures to show the attainment and progress of learners attracting the Pupil Premium.
  • Comparison of each school’s performance with that of schools with similar intakes.

None are mentioned here, nor are any of the suggestions advanced by respondents taken up.

Floor standards

Changes are proposed to the floor standards with effect from September 2016.

This section of the response begins by committing to:

‘…a new floor standard that holds schools to account both on the progress that they make and on how well their pupils achieve.’

But the plans set out subsequently do not meet this description.

The progress element of the current floor standard relates to any of reading, writing or mathematics but, under the new floor standard, it will relate to all three of these together.

An all-though primary school must demonstrate that:

‘…pupils make sufficient progress at key stage 2 from their starting point…’

As we have noted above, all-through primaries can opt to use the KS1 baseline or the Year R baseline in 2015. Moreover, from 2016 they can choose not to use the Year R baseline and be assessed solely on the attainment measure in the floor standards (see below).

Junior and middle schools obviously apply the KS1 baseline, while arrangements for infant and first schools have yet to be finalised.

What constitutes ‘sufficient progress’ is not defined. Annex C of the response says:

‘For 2016 we will set the precise extent of progress required once key stage 2 tests have been sat for the first time.’

Presumably this will be progress from KS1 to KS2, since progress from the Year R baseline will not be introduced until 2023.

The attainment element of the new floor standards is for schools to have 85% or more of pupils meeting the new, higher threshold standard at the end of KS2 in all of reading, writing and maths. The text says explicitly that this threshold is ‘similar to a level 4b under the current system’.

Annex C clarifies that this will be judged by the achievement of a scaled score of 100 or more in each of the reading and maths tests, plus teacher assessment that learners have reached the expected standard in writing (so the GPS test does not count in the same way, simply informing the teacher assessment).

As noted above, this a far bigger ask than the current reference to 65% of learners meeting the expected (and lower 4c) standard. The summary at the beginning of the response refers to it as ‘a challenging aspiration’:

‘Over time we expect more and more schools to achieve this standard.’

The statement in the first paragraph of this section of the response led us to believe that these two requirements – for progress and attainment respectively – would be combined, so that schools would be held account for both (unless, presumably, they exercised their right to opt out of the Year R baseline assessment).

But this is not the case. Schools need only achieve one or the other.

It follows that schools with a very high performing intake may exceed the floor standards on the basis of all-round high attainment alone, regardless of the progress made by their learners.

The reason for this provision is unclear, though one suspects that schools with an extremely high attaining intake, whether at Reception or Year 3, will be harder pressed to achieve sufficient progress, presumably because some ceiling effects come into play at the end of KS2.

This in turn might suggest that the planned tests do not have sufficient headroom for the highest attainers, even though they are supposed to provide similar challenge to level 6 and potentially extend beyond it.

Meanwhile, schools with less than stellar attainment results will be obliged to follow the progress route to jump the floor standard. This too will be demanding because all three domains will be in play.

There will have been some internal modelling undertaken to judge how many schools would be likely to fall short of the floor standards given these arrangements and it would be very useful to know these estimates, however unreliable they prove to be.

In their absence, one suspects that the majority of schools will be below the floor standards, at least initially. That of course materially changes the nature and purpose of the standards.

To Do List

The response and the draft specifications together contain a long list of work to be carried out over the next two years or so. I have included below my best guess as to the latest possible date for each decision to be completed and communicated:

  • Decide how progress will be measured for infants and first schools between the Year R baseline and the end of KS1 (April 2014)
  • Make available to schools a ‘small number’ of sample test questions for each key stage and subject (Summer 2014)
  • Work with experts to establish the criteria for the Year R baseline (September 2014)
  • KS1 [and KS2?] teacher assessment performance descriptors to be drafted by an expert group (September 2014)
  • Complete and report outcomes of a study with schools that already use Year R baseline assessments (December 2014)
  • Decide how Year R baseline assessments will be moderated (December 2014)
  • Publish a list of assessments that meet the Year R baseline criteria (March 2015)
  • Decide how Year R baseline results will be communicated to parents and to Ofsted (March 2015)
  • Make available to schools a full set of sample materials including tests and mark schemes for all KS1 and KS2 tests (September 2015)
  • Complete work with Ofsted and Teachers to improve KS1 moderation (September 2015)
  • Provide further information to enable teachers to assess pupils at the end of KS1 and KS2 who are ‘working above the P-scales but below the level of the test’ (September 2015)
  • Decide whether to move to external moderation of P-scale teacher assessment (September 2015)
  • Agree with stakeholders how to compare schools’ performance on a suite of assessment outcomes published in a standard format (September 2015)
  • Publish all final test frameworks (Autumn 2015)
  • Introduce new requirements for schools to publish a suite of assessment outcomes in a standard format (Spring 2016)
  • Panels of teacher use level descriptors to set the standards on the new tests following their first administration in May 2016 (Summer 2016)
  • Define what counts as sufficient progress from the Year R baseline to end KS1 and end KS2 respectively (Summer 2016)

Conclusion

Overall the response is rather more cogent and coherent than the original consultation document, though there are several inconsistencies and many sins of omission.

Drawing together the key issues emerging from the commentary above, I would highlight twelve key points:

  • The declared aims express the policy direction clumsily and without conviction. The ultimate aspirations are universal ‘secondary readiness’ (though expressed in broader terms), ‘no child left behind’ and ‘every child fulfilling their potential’ but there is no real effort to reconcile these potentially conflicting notions into a consensual vision of what primary education is for. Moreover, an inconvenient truth lurks behind these statements. By raising expectations so significantly – 4b equivalent rather than 4c; 85% over the attainment threshold rather than 65%; ‘sufficient progress’ rather than median progress and across three domains rather than one – there will be much more failure in the short to medium term. More learners will fall behind and fall short of the thresholds; many more schools are likely to undershoot the floor standards. It may also prove harder for some learners to demonstrate their potential. It might have been better to acknowledge this reality and to frame the vision in terms of creating the conditions necessary for subsequent progress towards the ultimate aspirations.
  • Younger children are increasingly caught in the crossbeam from the twin searchlights of assessment and accountability. HMCI’s subsequent intervention has raised the stakes still further. This creates obvious tensions in the sector which can be traced back to disagreements over the respective purposes of early years and primary provision and how they relate to each other. (HMCI’s notion of ‘school readiness’ is no doubt as narrow to early years practitioners as ‘secondary readiness’ is to primary educators.) But this is not just a theoretical point. Additional demands for focused inspection, moderation and publication of outcomes all carry a significant price tag. It must be open to question whether the sheer weight of assessment activity is optimal and delivers value for money. Should a radical future Government – probably with a cost-cutting remit – have rationalisation in mind?
  • Giving schools the freedom to choose from a range of Year R baseline assessment tools also seems inherently inefficient and flies in the face of the clear majority of consultation responses. We are told nothing of the perceived quality of existing services, none of which can – by definition – satisfy these new expectations without significant adjustment. It will not be straightforward to construct a universal and child-friendly instrument that is a sufficiently strong predictor of Level 4b-equivalent performance in KS2 reading, writing and maths assessments undertaken seven years later. Moreover, there will be a strong temptation for the Government to pitch the baseline higher than current expectations, so matching the  realignment at the other end of the process. Making the Reception baseline assessment optional – albeit with strings attached – seems rather half-hearted, almost an insurance against failure. Effective (and expensive) moderation may protect against widespread gaming, but the risk remains that Reception teachers will be even more predisposed to prioritise universal school readiness over stretching their more precocious four year-olds.
  • The task of designing an effective test for all levels of prior attainment at the end of key stage 2 is equally fraught with difficulty. The P-scales will be retained (in their existing format, unaligned with the revised national curriculum) for learners with special needs working below the equivalent of what is currently level 1. There will also be undefined provision ‘for those working above the level of the P-scales but below the level of the test’, even though the draft test development frameworks say:

‘All eligible children who are registered at maintained schools, special schools, or academies (including free schools) in England and are at the end of key stage 2 will be required to take the…test, unless they have taken it in the past.’

And this applies to all learners other than those in the exempted categories set out in the ARA booklets. The draft specifications add that test questions will be placed in order of difficulty. I have grave difficulty in understanding how such assessments can be optimal for high attainers and fear that this is bad assessment practice.

  • On top of this there is the worrying statement in the test development frameworks that scaled scores will be ‘truncated’ at the extremes of the distribution’. This does not fill one with confidence that the highest and lowest attainers will have their test performance properly recognised and reported.
  • The necessary invention of ‘performance descriptors’ removes any lingering illusion that academies and free schools have significant freedom to depart from the national curriculum, at least as far as the core subjects are concerned. It is hard to understand why these descriptors could not have been published alongside the programmes of study within the national curriculum.
  • The ‘performance descriptors’ in the draft test specifications carry all sorts of health warnings that they are inappropriate for teacher assessment because they cover only material that can be assessed in a written test. But there will be significant overlap between the test and teacher assessment versions, particularly in those that describe threshold performance at the equivalent of level 4b. For we know now that there will also be hierarchies of performance descriptors – aka level descriptors – for KS1 teacher assessment in reading, writing, speaking and listening and maths, as well as for KS2 teacher assessment in writing. Levels were so problematic that it has been necessary to reinvent them!
  • What with scaled scores, average scaled scores, threshold performance descriptors and ‘levelled’ performance descriptors, schools face an uphill battle in convincing parents that the reporting of test outcomes under this system will be simpler and more understandable. At the end of KS2 they will receive 16 different assessments in four different formats. (Remember that parents will also need to cope with schools’ approaches to internal assessment, which may or may not align with these arrangements.)
  • We are told about new requirements to be placed on schools to publish assessment outcomes, but the description is infuriatingly vague. We do not know whether certain requirements apply to both KS1 and 2, and/or to both tests and teacher assessment. The reference to ‘the percentage of pupils who achieve a high score in all areas at the end of key stage 2’ is additionally vague because it is unclear whether it applies to performance in each assessment, or across all assessments combined. Nor is the pitch of the high score explained. This is the only reference to high attainers in the entire response and it raises more questions than it answers.
  • We also have negligible information about what will appear in the school performance tables and what will be relegated to the accompanying data portal. We know there is an intention to compare schools’ performance on the measures they are required to publish and that is all. Much of the further detail in the original consultation document may or may not have fallen by the wayside.
  • The new floor standards have all the characteristics of a last-minute compromise hastily stitched together. The consultation document was explicit that floor standards would:

‘…focus on threshold attainment measures and value-added progress measures’

It anticipated that the progress measure would require average scaled scores of between 98.5 and 99.0 adding:

‘Our modelling suggests that a progress measure set at this level, combined with the 85% threshold attainment measure, would result in a similar number of schools falling below the floor as at present.’

But the analysis of responses fails to report at all on the question ‘Do you have any comments about these proposals for the Department’s floor standards?’ It does include the response to a subsequent question about including an average point score attainment measure in the floor standards (39% of respondents were in favour of this against 31% against). But the main text does not discuss this option at all. It begins by stating that both an attainment and a progress dimension are in play, but then describes a system in which schools can choose one or the other. There is no attempt to quantify ‘sufficient progress’ and no revised modelling of the impact of standards set at this level. We are left with the suspicion that a very significant proportion of schools will not exceed the floor. There is also a potential perverse incentive for schools with very high attaining intakes not to bother about progress at all.

  • Finally, the ‘to do’ list is substantial. Several of those with the tightest deadlines ought really to have been completed ahead of the consultation response, especially given the significant delay. There is nothing about the interaction between this work programme and that proposed by NAHT’s Commission on Assessment. Much of this work would need to take place on the other side of a General Election, while the lead time for assessing KS2 progress against a Year R baseline is a full nine years. This makes the project as a whole particularly vulnerable to the whims of future governments.

I’m struggling to find the right description for the overall package. I don’t think it’s quite substantial or messy enough to count as a dog’s breakfast. But, like a poorly airbrushed portrait, it flatters to deceive. Seen from a distance it appears convincing but, on closer inspection, there are too many wrinkles that have not been properly smoothed out

GP

April 2014

 

 

Challenging NAHT’s Commission on Assessment

.

This post reviews the Report of the NAHT’s National Commission on Assessment, published on 13 February 2014.

pencil-145970_640Since I previously subjected the Government’s consultation document on primary assessment and accountability to a forensic examination, I thought it only fair that I should apply the same high standards to this document.

I conclude that the Report is broadly helpful, but there are several internal inconsistencies and a few serious flaws.

Impatient readers may wish to skip the detailed analysis and jump straight to the summary at the end of the post which sets out my reservations in the form of 23 recommendations addressed to the Commission and the NAHT.

.

Other perspectives

Immediate reaction to the Report was almost entirely positive.

The TES included a brief Ministerial statement in its coverage, attributed to Michael Gove:

‘The NAHT’s report gives practical, helpful ideas to schools preparing for the removal of levels. It also encourages them to make the most of the freedom they now have to develop innovative approaches to assessment that meet the needs of pupils and give far more useful information to parents.’

ASCL and ATL both welcomed the Report, as did the National Governors’ Association, though there was no substantive comment from NASUWT or NUT.

The Blogosphere exhibited relatively little interest, although a smattering of posts began to expose some issues:

  • LKMco supported the key recommendations, but wondered whether the Commission might not be guilty of reinventing National Curriculum levels;
  • Mr Thomas Maths was more critical, identifying three key shortcomings, one being the proposed approach to differentiation within assessment;
  • Warwick Mansell, probably because he blogs for NAHT, confined himself largely to summarising the Report, which he found ‘impressive’, though he did raise two key points – the cost of implementing these proposals and how the recommendations relate to the as yet uncertain position of teacher assessment in the Government’s primary assessment and accountability reforms.

All of these points – and others – are fleshed out in the critique below.

.

Background

.

Remit, Membership and Evidence Base

The Commission was first announced in July 2013, when it was described as:

‘a commission of practitioners to shape the future of assessment in a system without levels.’

By September, Lord Sutherland had agreed to Chair the body and its broad remit had been established:

‘To:

  • establish a set of principles to underpin national approaches to assessment and create consistency;
  • identify and highlight examples of good practice; and
  • build confidence in the assessment system by securing the trust and support of officials and inspectors.’

Written evidence was requested by 16 October.

The first meeting took place on 21 October and five more were scheduled before the end of November.

Members’ names were not included at this stage (beyond the fact that NAHT’s President – a Staffordshire primary head – was involved) though membership was now described as ‘drawn from across education’.

Several members had in fact been named in an early October blog post from NAHT and a November press release from the Chartered Institute of Educational Assessors (CIEA) named all but one – NAHT’s Director of Education. This list was confirmed in the published Report.

The Commission had 14 members but only six of them – four primary heads one primary deputy and one secondary deputy – could be described as practitioners.

The others included two NAHT officials in addition to the secretariat, one being General Secretary Russell Hobby, and one from ASCL;  John Dunford, a consultant with several other strings to his bow, one of those being Chairmanship of the CIEA; Gordon Stobart an academic specialist in assessment with a long pedigree in the field; Hilary Emery, the outgoing Chief Executive of the National Children’s Bureau; and Sam Freedman of Teach First.

There were also unnamed observers from DfE, Ofqual and Ofsted.

The Report says the Commission took oral evidence from a wide range of sources. A list of 25 sources is provided but it does not indicate how much of their evidence was written and how much oral.

Three of these sources are bodies represented on the Commission, two of them schools. Overall seven are from schools. One source is Tim Oates, the former Chair of the National Curriculum Review Expert Panel.

The written evidence is not published and I could find only a handful of responses online, from:

Overall one has to say that the response to the call for evidence was rather limited. Nevertheless, it would be helpful for NAHT to publish all the evidence it received. It might be helpful for NAHT to consult formally on key provisions in its Report.

 .

Structure of the Report and Further Stages Proposed

The main body of the Report is sandwiched between a foreword by the Chair and a series of Annexes containing case studies, historical and international background.  This analysis concentrates almost entirely on the main body.

The 21 Recommendations are presented twice, first as a list within the Executive Summary and subsequently interspersed within a thematic commentary that summarises the evidence received and also conveys the Commission’s views.

The Executive Summary also sets out a series of Underpinning Principles for Assessment and a Design Checklist for assessment in schools, the latter accompanied by a set of five explanatory notes.

It offers a slightly different version of the Commission’s Remit:

‘In carrying out its task, the Commission was asked to achieve three distinct elements:

  • A set of agreed principles for good assessment
  • Examples of current best practice in assessment that meet these principles
  • Buy-in to the principles by those who hold schools to account.’

These are markedly less ambitious than their predecessors, having dropped the reference to ‘national approaches’ and any aspiration to secure support from officials and inspectors for anything beyond the Principles.

Significantly, the Report is presented as only the first stage in a longer process, an urgent response to schools’ need for guidance in the short term.

It recommends that further work should comprise:

  • ‘A set of model assessment criteria based on the new National Curriculum.’ (NAHT is called upon to develop and promote these. The text says that a model document is being  commissioned but doesn’t reveal the timescale or who is preparing it);
  • ‘A full model assessment policy and procedures, backed by appropriate professional development’ that would expand upon the Principles and Design Checklist. (NAHT is called upon to take the lead in this, but there is no indication that they plan to do so. No timescale is attached)
  • ‘A system-wide review of assessment’ covering ages 2-19. It is not explicitly stated, but one assumes that this recommendation is directed towards the Government. Again no timescale is attached.

The analysis below looks first at the assessment Principles, then the Design Checklist and finally the recommendations plus associated commentary. It concludes with an overall assessment of the Report as a whole.

.

Assessment Principles

As noted above, it seems that national level commitment is only sought in respect of these Principles, but there is no indication in the Report – or elsewhere for that matter – that DfE, Ofsted and Ofqual have indeed signed up to them.

Certainly the Ministerial statement quoted above stops well short of doing so.

The consultation document on primary assessment and accountability also sought comments on a set of core principles to underpin schools’ curriculum and assessment frameworks. It remains to be seen whether the version set out in the consultation response will match those advanced by the Commission.

The Report recommends that schools should review their own assessment practice against the Principles and Checklist together, and that all schools should have their own clear assessment principles, presumably derived or adjusted in the light of this process.

Many of the principles are unexceptionable, but there are a few interesting features that are directly relevant to the commentary below.

For it is of course critical to the internal coherence of the Report that the Design Checklist and recommendations are entirely consistent with these Principles.

I want to highlight three in particular:

  • ‘Assessment is inclusive of all abilities…Assessment embodies, through objective criteria, a pathway of progress and development for every child…Assessment objectives set high expectations for learners’.

One assumes that ‘abilities’ is intended to stand proxy for both attainment and potential, so that there should be ‘high expectations’ and a ‘pathway of progress and development’ for the lowest and highest attainers alike.

  • ‘Assessment places achievement in context against nationally standardised criteria and expected standards’.

This begs the question whether the ‘model document’ containing assessment criteria commissioned by NAHT will be ‘nationally standardised’ and, if so, what standardisation process will be applied.

  • ‘Assessment is consistent…The results are readily understandable by third parties…A school’s results are capable of comparison with other schools, both locally and nationally’.

The implication behind these statements must be that results of assessment in each school are transparent and comparable through the accountability regime, presumably by means of the performance tables (and the data portal that we expect to be introduced to support them).

This cannot be taken as confined to statutory tests, since the text later points out that:

‘The remit did not extend to KS2 tests, floor standards and other related issues of formal accountability.’

It isn’t clear, from the Principles at least, whether the Commission believes that teacher assessment outcomes should also be comparable. Here, as elsewhere, the Report does a poor job of distinguishing between statutory teacher assessment and assessment internal to the school.

.

Design Checklist.

 

Approach to Assessment and Use of Assessment

The Design Checklist is described as:

‘an evaluation checklist for schools seeking to develop or acquire an assessment system. They could also form the seed of a revised assessment policy.’

It is addressed explicitly to schools and comprises three sections covering, respectively, a school’s approach to assessment, method of assessment and use of assessment.

The middle section is by far the most significant and also the most complex, requiring five explanatory notes.

I have taken the more straightforward first and third sections first.

‘Our approach to assessment’ simply makes the point that assessment is integral to teaching and learning, while also setting expectations for regular, universal professional development and ‘a senior leader who is responsible for assessment’.

It is not clear whether this individual is the same as, or additional to, the ‘trained assessment lead’ mentioned in the Report’s recommendations.

I can find no justification in the Report for the requirement that this person must be a senior leader.

A more flexible approach would be preferable, in which the functions to be undertaken are outlined and schools are given flexibility over how those are distributed between staff. There is more on this below.

The final section ‘Our use of assessment’ refers to staff:

  • Summarising and analysing attainment and progress;
  • Planning pupils’ learning to ensure every pupil meets or exceeds expectations (Either this is a counsel of perfection, or expectations for some learners are pitched below the level required to satisfy the assessment criteria for the subject and year in question. The latter is much more likely, but this is confusing since satisfying the assessment criteria is also described in the Checklist in terms of ‘meeting…expectations’.)
  • Analysing data across the school to ensure all pupils are stretched while the vulnerable and those at risk make appropriate progress (‘appropriate’ is not defined within the Checklist itself but an explanatory note appended to the central section  – see below – glosses this phrase);
  • Communicating assessment information each term to pupils and parents through ‘a structured conversation’ and the provision of ‘rich, qualitative profiles of what has been achieved and indications of what they [ie parents as well as pupils] need to do next’; and
  • Celebrating a broad range of achievements, extending across the full school curriculum and encompassing social, emotional and behavioural development.

.

Method of Assessment: Purposes

‘Our method of assessment’ is by far the longest section, containing 11 separate bullet points. It could be further subdivided for clarity’s sake.

The first three bullets are devoted principally to some purposes of assessment. Some of this material might be placed more logically in the ‘Our Use of Assessment’ section, so that the central section is shortened and restricted to methodology.

The main purpose is stipulated as ‘to help teachers, parents and pupils plan their next steps in learning’.

So the phrasing suggests that assessment should help to drive forward the learning of parents and teachers, as well as to the learning of pupils. I’m not sure if this is deliberate or accidental.

Two subsidiary purposes are mentioned: providing a check on teaching standards and support for their improvement; and providing a comparator with other schools via collaboration and the use of ‘external tests and assessments’.

It is not clear why these three purposes are singled out. There is some overlap with the Principles but also a degree of inconsistency between the two pieces of documentation. It might have been better to cross-reference them more carefully.

In short, the internal logic of the Checklist and its relationship with the Principles could both do with some attention.

The real meat of the section is incorporated in the eight remaining bullet points. The first four are about what pupils are assessed against and when that assessment takes place. The last four explain how assessment judgements are differentiated, evidenced and moderated.

.

Method of Assessment: What Learners Are Assessed Against – and When

The next four bullets specify that learners are to be assessed against ‘assessment criteria which are short, discrete, qualitative and concrete descriptions of what a pupil is expected to know and be able to do.’

These are derived from the school curriculum ‘which is composed of the National Curriculum and our own local design’ (Of course that is not strictly the position in academies, as another section of the Report subsequently points out.)

The criteria ‘for periodic assessment are arranged into a hierarchy setting out what children are normally expected to have mastered by the end of each year’.

Each learner’s achievement ‘is assessed against all the relevant criteria at appropriate times of the school year’.

.

The Span of the Assessment Criteria

The first explanatory note (A) clarifies that the assessment criteria are ‘discrete, tangible descriptive statements of attainment’ derived from ‘the National Curriculum (and any school curricula)’.

There is no repetition of the provision in the Principles that they should be ‘nationally standardised’ but ‘there is little room for meaningful variety’, even though academies are not obliged to follow the National Curriculum and schools have complete flexibility over the remainder of the school curriculum.

The Recommendations have a different emphasis, saying that NAHT’s model criteria should be ‘based on the new National Curriculum’ (Recommendation 6), but the clear impression here is that they will encompass the National Curriculum ‘and any school curricula’ alike.

This inconsistency needs to be resolved. NAHT might be better off confining its model criteria to the National Curriculum only – and making it clear that even these may not be relevant to academies.

.

The Hierarchy of Assessment Criteria

The second explanatory note (B) relates to the arrangement of the assessment criteria

‘…into a hierarchy, setting out what children are normally expected to have mastered by the end of each year’.

This note is rather muddled.

It begins by suggesting that a hierarchy divided chronologically by school year is the most natural choice, because:

‘The curriculum is usually organised into years and terms for planned delivery’

That may be true, but only the Programmes of Study for the three core subjects are organised by year, and each clearly states that:

‘Schools are…only required to teach the relevant programme of study by the end of the key stage. Within each key stage, schools therefore have the flexibility to introduce content earlier or later than set out in the programme of study. In addition, schools can introduce key stage content during an earlier key stage if appropriate.’

All schools – academies and non-academies alike – therefore enjoy considerable flexibility over the distribution of the Programmes of Study between academic years.

(Later in the Report – in the commentary preceding the first six recommendations – the text mistakenly suggests that the entirety of ‘the revised curriculum is presented in a model of year-by-year progress’ (page 14) It does not mention the provision above).

The note goes on to suggest that the Commission has chosen a different route, not because of this flexibility, but because ‘children’s progress may not fit neatly into school years’:

‘…we have chosen the language of a hierarchy of expectations to avoid misunderstandings. Children may be working above or below their school year…’

But this is not an absolute hierarchy of expectations – in the sense that learners are free to progress entirely according to ability (or, more accurately, their prior attainment) rather than in age-related lock steps.

In a true hierarchy of expectations, learners would be able to progress as fast or as slowly as they are able to, within the boundaries set by:

  • On one hand, high expectations, commensurate challenge and progression;
  • On the other hand, protection against excessive pressure and hot-housing and a judicious blending of faster pace with more breadth and depth (of which more below).

This is no more than a hierarchy by school year with some limited flexibility at the margins.

.

The timing of assessment against the criteria

The third explanatory note (C) confirms the Commission’s assumption that formal assessments will be conducted at least termly – and possibly more frequently than that.

It adds:

‘It will take time before schools develop a sense of how many criteria from each year’s expectations are normally met in the autumn, spring and summer terms, and this will also vary by subject’.

This is again unclear. It could mean that a future aspiration is to judge progress termly, by breaking down the assessment criteria still further – so that a learner who met the assessment criteria for, say, the autumn term is deemed to be meeting the criteria for the year as a whole at that point.

Without this additional layer of lock-stepping, presumably the default position for the assessments conducted in the autumn and spring terms is that learners will still be working towards the assessment criteria for the year in question.

The note also mentions in passing that:

‘For some years to come, it will be hard to make predictions from outcomes of these assessments to the results in KS2 tests. Such data may emerge over time, although there are question marks over how reliable predictions may be if schools are using incompatible approaches and applying differing standards of performance and therefore cannot pool data to form large samples.’

This is one of very few places where the Report picks up on the problems that are likely to emerge from the dissonance between internal and external statutory assessment.

But it avoids the central issue, this being that the approach to internal assessment it advocates may not be entirely compatible with predicting future achievement in the KS2 tests. If so, its value is seriously diminished, both for parents and teachers, let alone the learners themselves.  This issue also reappears below.

.

Method of Assessment: How Assessment Judgements are Differentiated, Evidenced and Moderated

The four final bullet points in this section of the Design Checklist explain that all learners will be assessed as either ‘developing’, ‘meeting’, or ‘exceeding’ each relevant criterion for that year’.

Learners deemed to be exceeding the relevant criteria in a subject for a given year ‘will also be assessed against the criteria in that subject for the next year.’

Assessment judgements are supported by evidence comprising observations, records of work and test outcomes and are subject to moderation by teachers in the same school and in other schools to ensure they are fair, reliable and valid.

I will set moderation to one side until later in the post, since that too lies outside the scope of methodology.

.

Differentiation against the hierarchy of assessment criteria

The fourth explanatory note (D) addresses the vexed question of differentiation.

As readers may recall, the Report by the National Curriculum Review Expert Panel failed abjectly to explain how they would provide stretch and challenge in a system that focused exclusively on universal mastery and ‘readiness to progress’, saying only that further work was required to address the issue.

Paragraph 8.21 implied that they favoured what might be termed an ‘enrichment and extension’ model:

‘There are issues regarding ‘stretch and challenge’ for those pupils who, for a particular body of content, grasp material more swiftly than others. There are different responses to this in different national settings, but frequently there is a focus on additional activities that allow greater application and practice, additional topic study within the same area of content, and engagement in demonstration and discussion with others…These systems achieve comparatively low spread at the end of primary education, a factor vital in a high proportion of pupils being well positioned to make good use of more intensive subject-based provision in secondary schooling.’

Meanwhile, something akin to the P Scales might come into play for those children with learning difficulties.

On this latter point, the primary assessment and accountability consultation document said DfE would:

‘…explore whether P-scales should be reviewed so that they align with the revised national curriculum and provide a clear route to progress to higher attainment.’

We do not yet know whether this will happen, but Explanatory Note B to the Design Checklist conveys the clear message that the P-Scales need to be retained:

‘…must ensure we value the progress of children with special needs as much as any other group. The use of P scales here is important to ensure appropriate challenge and progression for pupils with SEN.’

By contrast, for high attainers, the Commission favours what might be called a ‘mildly accelerative’ model whereby learners who ‘exceed’ the assessment criteria applying to a subject for their year group may be given work that enables them to demonstrate progress against the criteria for the year above.

I describe it as mildly accelerative because there is no provision for learners to be assessed more than one year ahead of their chronological year group. This is a fairly low ceiling to impose on such accelerative progress.

It is also unclear whether the NAHT’s model assessment criteria will cover Year 7, the first year of the KS3 Programmes of Study, to enable this provision to extend into Year 6.

The optimal approach for high attainers would combine the ‘enrichment and extension’ approach apparently favoured by the Expert Panel with an accelerative approach that provides a higher ceiling, to accommodate those learners furthest ahead of their peers.

High attaining learners could then access a customised blend of enrichment (more breadth), extension (greater depth) and acceleration (faster pace) according to their needs.

This is good curricular practice and it should be reflected in assessment practice too, otherwise the risk is that a mildly accelerative assessment process will have an undesirable wash-back effect on teaching and learning.

Elsewhere, the Report advocates the important principle that curriculum, assessment and pedagogy should be developed in parallel, otherwise there is a risk that one – typically assessment – has an undesirable effect on the others. This would be an excellent exemplar of that statement.

The judgement whether a learner is exceeding the assessment criteria for their chronological year would be evidenced by enrichment and extension activity as well as by pre-empting the assessment criteria for the year ahead. Exceeding the criteria in terms of greater breadth or more depth should be equally valued.

This more rounded approach, incorporating a higher ceiling, should also be supported by the addition of a fourth ‘far exceeded’ judgement, otherwise the ‘exceeded’ judgement has to cover far too wide a span of attainment, from those who are marginally beyond their peers to those who are streets ahead.

These concerns need urgently to be addressed, before NAHT gets much further with its model criteria.

.

The aggregation of criteria

In order to make the overall judgement for each subject, learners’ performance against individual assessment criteria has to be combined to give an aggregate measure.

The note says:

‘The criteria themselves can be combined to provide the qualitative statement of a pupil’s achievements, although teachers and schools may need a quantitative summary. Few schools appear to favour a pure binary approach of yes/no. The most popular choice seems to be a three phase judgement of working towards (or emerging, developing), meeting (or mastered, confident, secure, expected) and exceeded. Where a student has exceeded a criterion, it may make sense to assess them also against the criteria for the next year.’

This, too, begs some questions. The statement above is consistent with one of the Report’s central recommendations:

‘Pupil progress and achievement should be communicated in terms of descriptive profiles rather than condensed to numerical summaries (although schools may wish to use numerical data for internal purposes).’

Frankly it seems unlikely that such ‘condensed numerical summaries’ can be kept hidden from parents. Indeed, one might argue that they have a reasonable right to know them.

These aggregations – whether qualitative or quantitative – will be differentiated at three levels, according to whether the learner best fits a ‘working towards’, ‘meeting’ or ‘exceeding’ judgement for the criteria relating to the appropriate year in each programme of study.

I have just recommended that there needs to be an additional level at the top end, to remove undesirable ceiling effects that lower expectations and are inconsistent with the Principles set out in the Report. I leave it to others to judge whether, if this was accepted, a fifth level is also required at the lower end to preserve the symmetry of the scale.

There is also a ‘chicken and egg’ issue here. It is not clear whether a learner must already be meeting some of the criteria for the succeeding year in order to show they are exceeding the criteria for their own year – or whether assessment against the criteria for the succeeding year is one potential consequence of a judgement that they are exceeding the criteria for their own year.

This confusion is reinforced by a difference of emphasis between the checklist – which says clearly that learners will be assessed against the criteria for the succeeding year if they exceeded the criteria for their own – and the explanatory note, which says only that this may happen.

Moreover, the note suggests that this applies criterion by criterion – ‘where a student has exceeded a criterion’ – rather than after the criteria have been aggregated, which is the logical assumption from the wording in the checklist – ‘exceeded the relevant criteria’.

This too needs clarifying.

.

.

Recommendations and Commentary

I will try not to repeat in this section material already covered above.

I found that the recommendations did not always sit logically with the preceding commentary, so I have departed from the subsections used in the Report, grouping the material into four broad sections: further methodological issues; in-school and school-to school support; national support; and phased implementation.

Each section leads with the relevant Recommendations and folds in additional relevant material from different sections of the commentary. I have repeated recommendations where they are relevant to more than one section.

.

Further methodological issues

Recommendation 4: Pupils should be assessed against objective criteria rather than ranked against each other

Recommendation 5: Pupil progress and achievements should be communicated in terms of descriptive profiles rather than condensed to numerical summaries (although schools may wish to use numerical data for internal purposes.

Recommendation 6: In respect of the National Curriculum, we believe it is valuable – to aid communication and comparison – for schools to be using consistent criteria for assessment. To this end, we call upon NAHT to develop and promote a set of model assessment criteria based on the new National Curriculum.

The commentary discusses the evolution of National Curriculum levels, including the use of sub-levels and their application to progress as well as achievement. In doing so, it summarises the arguments for and against the retention of levels.

In favour of retention:

  • The system of levels provides a common language used by schools to summarise attainment and progress;
  • It is argued (by some professionals) that parents have grown up with levels and have an adequate grasp of what they mean;
  • The numerical basis of levels was useful to schools in analysing and tracking the performance of large numbers of pupils;
  • The decision to remove levels was unexpected and caused concern within the profession, especially as it was also announced that being ‘secondary ready’ was to be associated with the achievement of Level 4B;
  • If levels are removed, they must be replaced by a different common language, or at least ‘an element of compatibility or common understanding’ should several different assessment systems emerge.

In favour of removal:

  • It is argued (by the Government) that levels are not understood by parents and other stakeholders;
  • The numerical basis of levels does not have the richness of a more rounded description of achievement. The important narrative behind the headline number is often lost through over-simplification.
  • There are adverse effects from labelling learners with levels.

The Commission is also clear that the Government places too great a reliance on tests, particularly for accountability purposes. This has narrowed the curriculum and resulted in ‘teaching to the test’.

It also creates other perverse incentives, including the inflation of assessment outcomes for performance management purposes or, conversely, the deflation of assessment outcomes to increase the rate of progress during the subsequent key stage.

Moreover, curriculum, assessment and pedagogy must be mutually supportive. Although the Government has not allowed the assessment tail to wag the curricular dog:

‘…curriculum and assessment should be developed in tandem.’

Self-evidently, this has not happened, since the National Curriculum was finalised way ahead of the associated assessment arrangements which, in the primary sector, are still unconfirmed.

There is a strong argument that such assessment criteria should have been developed by the Government and made integral to the National Curriculum.

Indeed, in Chapter 7 of its Report on ‘The Framework for the National Curriculum’, the National Curriculum Expert Panel proposed that attainment targets should be retained, not in the form of level descriptors but as ‘statements of specific learning outcomes related to essential knowledge’ that  would be ’both detailed and precise’. They might be presented alongside the Programmes of Study.

The Government ignored this, opting for a very broad single, standard attainment target in each programme of study:

‘By the end of each key stage, pupils are expected to know, apply and understand the matters, skills and processes specified in the relevant programme of study.’

As I pointed out in a previous post, one particularly glaring omission from the Consultation Document on Primary Assessment and Accountability was any explanation of how Key Stage Two tests and statutory teacher assessments would be developed from these singleton ‘lowest common denominator’ attainment targets, especially in a context where academies, while not obliged to follow the National Curriculum, would undertake the associated tests.

We must await the long-delayed response to the consultation to see if it throws any light on this matter.

Will it commit the Government to producing a framework, at least for statutory tests in the core subjects, or will it throw its weight behind the NAHT’s model criteria instead?

I have summarised this section of the Report in some detail as it is the nearest it gets to providing a rational justification for the approach set out in the recommendations above.

The model criteria appear confined to the National Curriculum at this point, though we have already noted that is not the case elsewhere in the Report.

I have also discussed briefly the inconsistency in permitting the translation of descriptive profiles into numerical data ‘for internal purposes’, but undertook to develop that further, for there is a wider case that the Report does not entertain.

We know that there will be scores attached to KS2 tests, since those are needed to inform parents and for accountability purposes.

The Primary Assessment and Accountability consultation document proposed a tripartite approach:

  • Scaled scores to show attainment, built around a new ‘secondary-ready’ standard, broadly comparable with the current Level 4B;
  • Allocation to a decile within the range of scaled scores achieved nationally, showing attainment compared with one’s peers; and
  • Comparison with the average scaled score of those nationally with the same prior attainment at the baseline, to show relative progress.

Crudely speaking, the first of these measures is criterion-referenced while the second and third are norm-referenced.

We do not yet know whether these proposals will proceed – there has been some suggestion that deciles at least will be dropped – but parents will undoubtedly want schools to be able to tell them what scaled scores their children are on target to achieve, and how those compare with the average for those with similar prior attainment.

It will be exceptionally difficult for schools to convey that information within the descriptive profiles, insofar as they relate to English and maths, without adopting the same numerical measures.

It might be more helpful to schools if the NAHT’s recommendations recognised that fact. For the brutal truth is that, if schools’ internal assessment processes do not respond to this need, they will have to set up parallel processes that do so.

In order to derive descriptive profiles, there must be objective assessment criteria that supply the building blocks, hence the first part of Recommendation 4. But I can find nothing in the Report that explains explicitly why pupils cannot also be ranked against each other. This can only be a veiled and unsubstantiated objection to deciles.

Of course it would be quite possible to rank pupils at school level and, in effect, that is what schools will do when they condense the descriptive profiles into numerical summaries.

The real position here is that such rankings would exist, but would not be communicated to parents, for fear of ‘labelling’. But the labelling has already occurred, so the resistance is attributable solely to communicating these numerical outcomes to parents. That is not a sustainable position.

.

In-school and school-to-school support

Recommendation 1: Schools should review their assessment practice against the principles and checklist set out in this report. Staff should be involved in the evaluation of existing practice and the development of a new, rigorous assessment system and procedures to enable the school to promote high quality teaching and learning.

Recommendation 2: All schools should have clear assessment principles and practices to which all staff are committed and which are implemented. These principles should be supported by school governors and accessible to parents, other stakeholders and the wider school community.

Recommendation 3: Assessment should be part of all school development plans and should be reviewed regularly. This review process should involve every school identifying its own learning and development needs for assessment. Schools should allocate specific time and resources for professional development in this area and should monitor how the identified needs are being met.

Recommendation 7 (part): Schools should work in collaboration, for example in clusters, to ensure a consistent approach to assessment. Furthermore, excellent practice in assessment should be identified and publicised…

Recommendation 9: Schools should identify a trained assessment lead, who will work with other local leads and nationally accredited assessment experts on moderation activities.

Recommendation 16: All those responsible for children’s learning should undertake rigorous training in formative, diagnostic and summative assessment, which covers how assessment can be used to support teaching and learning for all pupils, including those with special educational needs. The government should provide support and resources for accredited training for school assessment leads and schools should make assessment training a priority.

Recommendation 20: Schools should be asked to publish their principles of assessment from September 2014, rather than being required to publish a detailed assessment framework, which instead should be published by 2016. The development of the full framework should be outlined in the school development plan with appropriate milestones that allow the school sufficient time to develop an effective model.

All these recommendations are perfectly reasonable in themselves, but it is worth reflecting for a while on the likely cost and workload implications, particularly for smaller primary schools:

Each school must have a ‘trained assessment lead’ who may or may not be the same as the ‘senior leader who is responsible for assessment’ mentioned in the Design Checklist. There is no list of responsibilities for that person, but it would presumably include:

  • Leading the review of assessment practice and developing a new assessment system;
  • Leading the definition of the school’s assessment principles and practices and communicating these to governors, parents, stakeholders and the wider community;
  • Lead responsibility for the coverage of assessment within the school’s development plan and the regular review of that coverage;
  • Leading the identification and monitoring of the school’s learning and development needs for assessment;
  • Ensuring that all staff receive appropriate professional development – including ‘rigorous training in formative diagnostic and summative assessment’;
  • Leading the provision of in-school and school-to-school professional development relating to assessment;
  • Allocating time and resources for all assessment-related professional development and monitoring its impact;
  • Leading collaborative work with other schools to ensure a consistent approach to assessment;
  • Dissemination of effective practice;
  • Working with other local assessment leads and external assessment experts on moderation activities.

And, on top of this, there is a range of unspecified additional responsibilities associated with the statutory tests.

It is highly unlikely that this range of responsibilities could be undertaken effectively by a single person in less than half a day a week, as a bare minimum. There will also be periods of more intense pressure when a substantially larger time allocation is essential.

The corresponding salary cost for a ‘senior leader’ might be £3,000-£4,000 per year, not to mention the cost of undertaking the other responsibilities displaced.

There will also need to be a sizeable school budget and time allocation for staff to undertake reviews, professional development and moderation activities.

Moderation itself will bear a significant cost. Internal moderation may have a bigger opportunity cost but external moderation will otherwise be more expensive.

Explanatory note (E), attached to the Design Checklist, says:

‘The exact form of moderation will vary from school to school and from subject to subject. The majority of moderation (in schools large enough to support it) will be internal but all schools should undertake a proportion of external moderation each year, working with partner schools and local agencies.’

Hence the cost of external moderation will fall disproportionately on smaller schools with smaller budgets.

It would be wrong to suggest that this workload is completely new. To some extent these various responsibilities will be undertaken already, but the Commission’s recommendations are effectively a ratcheting up of the demand on schools.

Rather than insisting on these responsibilities being allocated to a single individual with other senior management responsibilities, it might be preferable to set out the responsibilities in more detail and give schools greater flexibility over how they should be distributed between staff.

Some of these tasks might require senior management input, but others could be handled by other staff, including paraprofessionals.

.

National support

Recommendation 7 (part): Furthermore, excellent practice in assessment should be identified and publicised, with the Department for Education responsible for ensuring that this is undertaken.

Recommendation 8 (part): Schools should be prepared to submit their assessment to external moderators, who should have the right to provide a written report to the head teacher and governors setting out a judgement on the quality and reliability of assessment in the school, on which the school should act. The Commission is of the view that at least some external moderation should be undertaken by moderators with no vested interest in the outcomes of the school’s assessment. This will avoid any conflicts of interest and provide objective scrutiny and broader alignment of standards across schools.

Recommendation 9: Schools should identify a trained assessment lead, who will work with other local leads and nationally accredited assessment experts on moderation activities.

Recommendation 11: The Ofsted school inspection framework should explore whether schools have effective assessment systems in place and consider how effectively schools are using pupil assessment information and data to improve learning in the classroom and at key points of transition between key stages and schools.

Recommendation 14: Further work should be undertaken to improve training for assessment within initial teacher training (ITT), the newly qualified teacher (NQT) induction year and on-going professional development. This will help to build assessment capacity and support a process of continual strengthening of practice within the school system.

Recommendation 15: The Universities’ Council for the Education of Teachers (UCET) should build provision in initial teacher training for delivery of the essential assessment knowledge.

Recommendation 16: All those responsible for children’s learning should undertake rigorous training in formative, diagnostic and summative assessment, which covers how assessment can be used to support teaching and learning for all pupils, including those with special educational needs. The government should provide support and resources for accredited training for school assessment leads and schools should make assessment training a priority.

Recommendation 17: A number of pilot studies should be undertaken to look at the use of information technology (IT) to support and broaden understanding and application of assessment practice.

Recommendation 19: To assist schools in developing a robust framework and language for assessment, we call upon the NAHT to take the lead in expanding the principles and design checklist contained in this report into a full model assessment policy and procedures, backed by appropriate professional development.

There are also several additional proposals in the commentary that do not make it into the formal recommendations:

  • Schools should be held accountable for the quality of their assessment practice as well as their assessment results, with headteachers also appraising teachers on their use of assessment. (The first part of this formulation appears in Recommendation 11 but not the second.) (p17);
  • It could be useful for the teaching standards to reflect further assessment knowledge, skills and understanding (p17);
  • A national standard in assessment practice for teachers would be a useful addition (p18);
  • The Commission also favoured the approach of having a lead assessor to work with each school or possibly a group of schools, helping to embed good practice across the profession (p18).

We need to take stock of the sheer scale of the infrastructure that is being proposed and its likely cost.

In respect of moderation alone, the Report is calling for sufficient external moderators, ‘nationally accredited assessment experts’ and possibly lead assessors to service some 17,000 primary schools.

Even if we assume that these roles are combined in the same person and that each person can service, say, 25 schools, that still demands something approaching a cadre of 700 people who also need to be supported, managed and trained.

If they are serving teachers there is an obvious opportunity cost. Providing a service of this scale would cost tens of millions of pounds a year.

Turning to training and professional development, the Commission is proposing:

  • Accredited training for some 17,000 school assessment leads (with an ongoing requirement to train new appointees and refresh the training of those who undertook it too far in the past);
  • ‘Rigorous training in formative, diagnostic and summative assessment, which covers how assessment can be used to support teaching and learning for all pupils, including those with special educational needs’ for everyone deemed responsible for children’s learning, so not just teachers. This will include hundreds of thousands of people in the primary sector alone.
  • Revitalised coverage of assessment in ITE and induction, on top of the requisite professional development package.

The Report says nothing of the cost of developing, providing and managing this huge training programme, which would cost some more tens of millions of pounds a year.

I am plucking a figure out of the air, but it would be reasonable to suggest that moderation and training costs combined might require an annual budget of some £50 million – and quite possibly double that. 

Unless one argues that the testing regime should be replaced by a national sampling process – and while the Report says some of the Commission’s members supported that, it stops short of recommending it – there are no obvious offsetting savings.

It is disappointing that the Commission made no effort at all to quantify the cost of its proposals.

These recommendations provide an excellent marketing opportunity for some of the bodies represented on the Commission.

For example, the CIEA press release welcoming the Report says:

‘One of the challenges, and one that schools will need to meet, is in working together, and with local and national assessment experts, to moderate their judgements and ensure they are working to common standards across the country. The CIEA has an important role to play in training these experts.’

Responsibility for undertaking pilot studies on the role of IT in assessment is not allocated, but one assumes it would be overseen by central government and also funded by the taxpayer.

Any rollout from the pilots would have additional costs attached and would more than likely create additional demand for professional development.

The reference to DfE taking responsibility for sharing excellent practice is already a commitment in the consultation document:

‘…we will provide examples of good practice which schools may wish to follow. We will work with professional associations, subject experts, education publishers and external test developers to signpost schools to a range of potential approaches.’ (paragraph 3.8).

Revision of the School Inspection Framework will require schools to give due priority to the quality of their assessment practice, though Ofsted might reasonably argue that it is already there.

Paragraph 116 of the School Inspection Handbook says:

‘Evidence gathered by inspectors during the course of the inspection should include… the quality and rigour of assessment, particularly in nursery, reception and Key Stage 1.’

We do not yet know whether NAHT will respond positively to the recommendation that it should go beyond the model assessment criteria it has already commissioned by leading work to expand the Principles and Design Checklist into ‘a full model assessment policy and procedures backed by appropriate professional development’.

There was no reference to such plans in the press release accompanying the Report.

Maybe the decision could not be ratified in time by the Association’s decision-making machinery – but this did not prevent the immediate commissioning of the model criteria.

.

Phased Implementation

Recommendation 10: Ofsted should articulate clearly how inspectors will take account of assessment practice in making judgements and ensure both guidance and training for inspectors is consistent with this.

Recommendation 12: The Department for Education should make a clear and unambiguous statement on the teacher assessment data that schools will be required to report to parents and submit to the Department for Education. Local authorities and other employers should provide similar clarity about requirements in their area of accountability.

Recommendation 13: The education system is entering a period of significant change in curriculum and assessment, where schools will be creating, testing and revising their policies and procedures. The government should make clear how they will take this into consideration when reviewing the way they hold schools accountable as new national assessment arrangements are introduced during 2014/15. Conclusions about trends in performance may not be robust.

Recommendation 18: The use by schools of suitably modified National Curriculum levels as an interim measure in 2014 should be supported by the government. However, schools need to be clear that any use of levels in relation to the new curriculum can only be a temporary arrangement to enable them to develop, implement and embed a robust new framework for assessment. Schools need to be conscious that the new curriculum is not in alignment with the old National Curriculum levels.

Recommendation 20: Schools should be asked to publish their principles of assessment from September 2014, rather than being required to publish a detailed assessment framework, which instead should be published by 2016. The development of the full framework should be outlined in the school development plan with appropriate milestones that allow the school sufficient time to develop an effective model.

Recommendation 21: A system wide review of assessment should be undertaken. This would help to repair the disjointed nature of assessment through all ages, 2-19.

The Commission quite rightly identifies a number of issues caused by the implementation timetable, combined with continuing uncertainty over aspects of the Government’s plans.

At the time of writing, the response to the consultation document has still not been published (though it was due in autumn 2013) yet schools will be implementing the new National Curriculum from this September.

The Report says:

‘There was strong concern expressed about the requirement for schools to publish their detailed curriculum and assessment framework in September 2014.’

This is repeated in Recommendation 20, together with the suggestion that this timeline should be amended so that only a school’s principles for assessment need be published by this September.

I have been trying to pin down the source of this requirement.

Schedule 4 of The School Information (England) (Amendment) Regulations 2012 do not require the publication of a detailed assessment framework, referring only to

‘The following information about the school curriculum—

(a)  in relation to each academic year, the content of the curriculum followed by the school for each subject and details as to how additional information relating to the curriculum may be obtained;

(b)  in relation to key stage 1, the names of any phonics or reading schemes in operation; and

(c)  in relation to key stage 4—

(i)            a list of the courses provided which lead to a GCSE qualification,

(ii)          a list of other courses offered at key stage 4 and the qualifications that may be acquired.’

I could find no Government guidance stating unequivocally that this requires schools to carve up all the National Curriculum programmes of study into year-by-year chunks.  (Though there is no additional burden attached to publication if they have already undertaken this task for planning purposes.)

There are references to the publication of Key Stage 2 results (which will presumably need updating to reflect the removal of levels), but nothing on the assessment framework.

Moreover, the DfE mandatory timeline says that from the Spring Term of 2014:

‘All schools must publish their school curriculum by subject and academic year, including their provision of personal, social, health and economic education (PSHE).’

(The hyperlink returns one to the Regulations quoted above.)

There is no requirement for publication of further information in September.

I wonder therefore if this is a misunderstanding. I stand to be corrected if readers can point me to the source.

It may arise from the primary assessment and accountability consultation document, which discusses publication of curricular details and then proceeds immediately to discuss the relationship between curriculum and assessment:

‘Schools are required to publish this curriculum on their website…In turn schools will be free to design their approaches to assessment, to support pupil attainment and progression. The assessment framework must be built into the school curriculum, so that schools can check what pupils have learned and whether they are on track to meet expectations at the end of the key stage, and so that they can report regularly to parents.’ (paras 3.4-3.5)

But this conflation isn’t supported by the evidence above and, anyway, these are merely proposals.

That said, it must be assumed that the Commission consulted its DfE observer on this point before basing recommendations on this interpretation.

If the observer’s response was consistent with the Commission’s interpretation, then it is apparently inconsistent with all the material so far published by the Department!

It may be necessary for NAHT to obtain clarification of this point given the evidence cited above.

That aside, there are issues associated with the transition from the current system to the future system.

The DfE’s January 2014 ‘myths and facts’ publication says:

‘As part of our reforms to the national curriculum, the current system of “levels” used to report children’s attainment and progress will be removed from September 2014. Levels are not being banned, but will not be updated to reflect the new national curriculum and will not be used to report the results of national curriculum tests. Key Stage 1 and Key Stage KS2 [sic] tests taken in the 2014 to 2015 academic year will be against the previous national curriculum, and will continue to use levels for reporting purposes

Schools will be expected to have in place approaches to formative assessment that support pupil attainment and progression. The assessment framework should be built into the school curriculum, so that schools can check what pupils have learned and whether they are on track to meet expectations at the end of the key stage, and so that they can report regularly to parents. Schools will have the flexibility to use approaches that work for their pupils and circumstances, without being constrained by a single national approach.’

The reference here to having approaches in place – rather than the publication of a ‘detailed curriculum and assessment framework’ – would not seem wildly inconsistent with the Commission’s idea that schools should establish their principles by September 2014, and develop their detailed assessment frameworks iteratively over the two succeeding years. However, the Government needs to clarify the position.

Since Key Stage 2 tests will not dispense with levels until May 2016 (and they will be published in the December 2015 Performance Tables), there will be an extended interregnum in which National Curriculum Levels will continue to have official currency.

Moreover, levels may still be used in schools – they are not being banned – though they will not be aligned to the new National Curriculum.

The Report says:

‘…it is important to recognise that, even if schools decide to continue with some form of levels, the new National Curriculum does not align to the existing levels and level descriptors and this alignment is a piece of work that needs to be undertaken now.’ (p19).

However, the undertaking of this work does not feature in the Recommendations, unless it is implicit in the production by NAHT of ‘a full model assessment policy and procedures’, which seems unlikely.

One suspects that the Government would be unwilling to endorse such a process, even as a temporary arrangement, since what is to stop schools from continuing to use this new improved levels structure more permanently?

The Commission would appear to be on stronger ground in asking Ofsted to make allowances during the interregnum (which is what I think Recommendation 10 is about) especially given that, as Recommendation 13 points out, evidence of ‘trends in performance may not be robust’.

The point about clarity over teacher assessment is well made – and one hopes it will form part of the response to the primary assessment and accountability consultation document when that is eventually published.

The Report itself could have made progress in this direction by establishing and maintaining a clearer distinction between statutory and internal teacher assessment.

The consultation document itself made clear that KS2 writing would continue to be assessed via teacher assessment rather than a test, and, moreover:

‘At the end of each key stage schools are required to report teacher assessment judgements in all national curriculum subjects to parents. Teachers will judge whether each pupil has met the expectations set out in the new national curriculum. We propose to continue publishing this teacher assessment in English, mathematics and science, as Lord Bew recommended.’ (para 3.9)

But what it does not say is what requirements will be imposed to ensure consistency across this data. Aside from KS2 writing, will they also be subject to the new scaled scores, and potentially deciles too?

Until schools have answers to that question, they cannot consider the overall shape of their assessment processes.

The final recommendation, for a system-wide review of assessment from 2-19 is whistling in the wind, especially given the level of disruption already caused by the decision to remove levels.

Neither this Government nor the next is likely to act upon it.

 

Conclusion

The Commission’s Report moves us forward in broadly the right direction.

The Principles, Design Checklist and wider recommendations help to fill some of the void created by the decision to remove National Curriculum levels, the limited nature of the primary assessment and accountability consultation document and the inordinate delay in the Government’s response to that consultation.

We are in a significantly better place as a consequence of this work being undertaken.

But there are some worrying inconsistencies in the Report as well as some significant shortcomings to the proposals it contains. There are also several unanswered questions.

Not to be outdone, I have bound these up into a series of recommendations directed at NAHT and its Commission. There are 23 in all and I have given mine letters rather than numerals, to distinguish them from the Commission’s own recommendations.

  • Recommendation A: The Commission should publish all the written evidence it received.
  • Recommendation B: The Commission should consult on key provisions within the Report, seeking explicit commitment to the Principles from DfE, Ofqual and Ofsted.
  •  Recommendation C: The Commission should ensure that its Design Checklist is fully consistent with the Principles in all respects. It should also revisit the internal logic of the Design Checklist.
  • Recommendation D: So far as possible, ahead of the primary assessment and accountability consultation response, the Commission should distinguish clearly how its proposals relate to statutory teacher assessment, alongside schools’ internal assessment processes.
  • Recommendation E: NAHT should confirm who it has commissioned to produce model assessment criteria and to what timetable. It should also explain how these criteria will be ‘nationally standardised’.
  • Recommendation F: The Commission should clarify whether the trained assessment lead mentioned in Recommendation 9 is the same or different to the ‘senior leader who is responsible for assessment’ mentioned in the Design Checklist.
  • Recommendation G: The Commission should set out more fully the responsibilities allocated to this role or roles and clarify that schools have flexibility over how they distribute those responsibilities between staff.
  • Recommendation H:  NAHT should clarify how the model criteria under development apply – if at all – to the wider school curriculum in all schools and to academies not following the National Curriculum.
  • Recommendation I: NAHT should clarify how the model criteria under development will allow for the fact that in all subjects all schools enjoy flexibility over the positioning of content in different years within the same key stage – and can also anticipate parts of the subsequent key stage.
  • Recommendation J: NAHT should clarify whether the intention is that the model criteria should reflect the allocation of content to specific terms as well as to specific years.
  • Recommendation K: The Commission should explain how its approach to internal assessment will help predict future performance in end of Key Stage tests.
  • Recommendation L: The Commission should shift from its narrow and ‘mildly accelerative’ view of high attainment to accommodate a richer concept that combines enrichment (breadth), extension (depth) and acceleration (faster pace) according to learners’ individual needs.
  • Recommendation M: The Commission should incorporate a fourth ‘far exceeded’ assessment judgement, since the ‘exceeded’ judgement covers too wide a span of attainment.
  • Recommendation N: NAHT should clarify whether its model criteria will extend into KS3, to accommodate assessment against the criteria for at least year 7, and ideally beyond.
  • Recommendation O: The Commission should clarify whether anticipating criteria for a subsequent year is a cause or a consequence of being judged to be ‘exceeding’ expectations in the learner’s own chronological year.
  • Recommendation P: The Commission should confirm that numerical summaries of assessment criteria – as well as any associated ranking positions – should be made available to parents who request them.
  • Recommendation Q: The Commission should explain why schools should be forbidden from ranking learners against each other (or allocating them to deciles).
  • Recommendation R: The Commission should assess the financial impact of its proposals on schools of different sizes.
  • Recommendation S: The Commission should cost its proposals for training and moderation, identifying the burden on the taxpayer and any offsetting savings.
  • Recommendation T: NAHT should clarify its response to Recommendation 19, that it should lead the development of a full model assessment policy and procedures.
  • Recommendation U: The Commission should clarify with DfE its understanding that schools are required to publish a detailed curriculum and assessment framework by September 2014.
  • Recommendation V: The Commission should clarify with DfE the expectation that it should have in place ‘approaches to formative assessment’ and whether the proposed assessment principles satisfy this requirement.
  • Recommendation W: The commission should clarify whether it is proposing that work is undertaken to align National Curriculum levels with the new National Curriculum and, if so, who it proposes should undertake this.

So – good overall – subject to these 23 reservations!

Some are more significant than others. Given my area of specialism, I feel particularly strongly about those that relate directly to high attainers, especially L and M above.

Those are the two I would nail to the door of 1 Heath Square.

.

GP

March 2014

What Becomes of Schools That Fail Their High Attainers?*

.

This post reviews the performance and subsequent history of schools with particularly poor results for high attainers in the Secondary School Performance Tables over the last three years.

P1010120

Seahorse in Perth Aquarium by Gifted Phoenix

It establishes a high attainer ‘floor target’ so as to draw a manageable sample of poor performers and, having done so:

  • Analyses the characteristics of this sample;
  • Explores whether these schools typically record poor performance in subsequent years or manage to rectify matters;
  • Examines the impact of various interventions, including falling below the official floor targets, being placed in special measures or deemed to have serious weaknesses following inspection, becoming an academy and receiving a pre-warning and/or warning notice;
  • Considers whether the most recent Ofsted reports on these schools do full justice to this issue, including those undertaken after September 2013 when new emphasis was placed on the performance of the ‘most able’.

The post builds on my previous analysis of high attainment in the 2013 School Performance Tables (January 2014). It applies the broad definition of high attainers used in the Tables, which I discussed in that post and have not repeated here.

I must emphasise at the outset that factors other than poor performance may partially explain particularly low scores in the Tables.

There may be several extenuating circumstances that are not reflected in the results. Sometimes these may surface in Ofsted inspection reports, but the accountability and school improvement regime typically imposes a degree of rough justice, and I have followed its lead.

It is also worth noting that the Performance Tables do not provide data for schools where the number of high attainers is five or fewer, because of the risk that individuals may be identifiable even though the data is anonymised.

This is unfortunate since the chances are that schools with very few high attainers will find it more difficult than others to address their needs. We may never know, but there is more on the impact of cohort size below.

Finally please accept my customary apology for any transcription errors. Do let me know if you notice any and I will correct them.

.

Drawing the Sample

The obvious solution would be to apply the existing floor targets to high attainers.

So it would include all schools recording:

  • Fewer than 35% (2011) or 40% (2012 and 2013) of high attainers achieving five or more GCSEs at grades A*-C or equivalent including GCSEs in English and mathematics and
  • Below median scores for the percentage of high attainers making at least the expected three levels of progress between Key Stages 2 and 4 in English and maths respectively.

But the first element is far too undemanding a threshold to apply for high attaining learners and the overall target generates a tiny sample.

The only school failing to achieve it in 2013 was Ark Kings Academy in Birmingham, which recorded just six high attainers, forming 9% of the cohort (so only just above the level at which results would have been suppressed).

In 2012 two schools were in the same boat:

  • The Rushden Community College in Northamptonshire, with 35 high attainers (26% of the cohort), which became a sponsored academy with the same name on 1 December 2012; and
  • Culverhay School in Bath and North East Somerset, with 10 high attainers (19% of the cohort), which became Bath Community Academy on 1 September 2012.

No schools at all performed at this level in 2011.

A sample of just three schools is rather too unrepresentative, so it is necessary to set a more demanding benchmark which combines the same threshold and progress elements.

The problem is not with the progress measure. Far too many schools fail to meet the median level of performance – around 70% each year in both English and maths – even with their cadres of high attainers. Hence I need to lower the pitch of this element to create a manageable sample.

I plumped for 60% or fewer high attainers making at least the expected progress between KS2 and KS4 in both English and maths. This captured 22 state-funded schools in 2013, 31 in 2012 and 38 in 2011. (It also enabled Ark King’s Academy to escape, by virtue of the fact that 67% of its high attainers learners achieved the requisite progress in English.)

For the threshold element I opted for 70% or fewer high attainers achieving five or more GCSEs at grades A*-C or equivalent including GCSEs in English and maths. This captured 19 state-funded schools in 2013, 29 in 2012 and 13 in 2011.

.

Venn 2.

The numbers of state-funded schools that met both criteria were seven in 2013, eight  in 2012 and five in 2011, so 20 in all.

I decided to feature this small group of schools in the present post while also keeping in mind the schools occupying each side of the Venn Diagram. I particularly wanted to see whether schools which emerged from the central sample in subsequent years continued to fall short on one or other of the constituent elements.

The 20 schools in the main sample are:

Table 1 below provides more detail about these 20 schools.

.

Table 1: Schools Falling Below Illustrative High Attainer Floor Targets 2011-2013

Name Type LA Status/Sponsor Subsequent History
2011
Carter Community School 12-16 mixed modern Poole Community Sponsored academy (ULT) 1/4/13
Hadden Park High School 11-16 mixed comp Nottingham Foundation Sponsored Academy (Bluecoat School) 1/1/14
Merchants Academy 11-18 mixed comp Bristol Sponsored Academy (Merchant Venturers/ University of Bristol
The Robert Napier School 11-18 mixed modern Medway Foundation Sponsored Academy (Fort Pitt Grammar School)  1/9/12
Bishop of Rochester Academy 11-18 mixed comp Kent Sponsored Academy (Medway Council/ Canterbury Christ Church University/ Diocese of Rochester)
2012
The Rushden Community College 11-18 mixed comp Northants Community Sponsored Academy (The Education Fellowship) 12/12
Culverhay School 11-18 boys comp Bath and NE Somerset Community Bath Community Academy – mixed (Cabot Learning) 1/9/12
Raincliffe School 11-16 mixed comp N Yorks Community Closed 8/12 (merged with Graham School)
The Coseley School 11-16 mixed comp Dudley Foundation
Fleetwood High School 11-18 mixed comp Lancs Foundation
John Spendluffe Foundation Technology College 11-16 mixed modern Lincs Academy converter
Parklands High School 11-18 mixed Liverpool Foundation Discussing academy sponsorship (Bright Tribe)
Frank F Harrison Engineering College 11-18 mixed comp Walsall Foundation Mirus Academy (sponsored by Walsall College) 1/1/12
2013
Gloucester Academy 11-19 mixed comp Glos Sponsored Academy (Prospect Education/ Gloucestershire College)
Christ the King Catholic and Church of England VA School 11-16 mixed comp Knowsley VA Closed 31/8/13
Aireville School 11-16 mixed modern N Yorks Community
Manchester Creative and Media Academy for Boys 11-19 boys comp Manchester Sponsored Academy (Manchester College/ Manchester Council/ Microsoft)
Fearns Community Sports College 11-16 mixed comp Lancs Community
Unity College Blackpool 5-16 mixed comp Blackpool Community Unity Academy Blackpool (sponsored by Fylde Coast Academies)
The Mirus Academy 3-19 mixed comp Walsall Sponsored Academy (Walsall College)

 .

Only one school appears twice over the three-year period albeit in two separate guises – Frank F Harrison/Mirus.

Of the 20 in the sample, seven were recorded in the relevant year’s Performance Tables as community schools, six as foundation schools, one was VA, one was an academy converter and the five remaining were sponsored academies.

Of the 14 that were not originally academies, seven have since become sponsored academies and one is discussing the prospect. Two more have closed, so just five – 25% of the sample – remain outside the academies sector.

All but two of the schools are mixed (the other two are boys’ schools). Four are modern schools and the remainder comprehensive.

Geographically they are concentrated in the Midlands and the North, with a few in the South-West and the extreme South-East. There are no representatives from London, the East or the North-East.

.

Performance of the Core Sample

Table 2 below looks at key Performance Table results for these schools. I have retained the separation by year and the order in which the schools appear, which reflects their performance on the GCSE threshold measure, with the poorest performing at the top of each section.

.

Table 2: Performance of schools falling below proposed high attainer floor targets 2011-2013

Name No of HA % HA 5+ A*-C incl E+M 3+ LoP En 3+ LoP Ma APS (GCSE)
2011
Carter Community School 9 13 56 56 44 304.9
Hadden Park High School 15 13 60 40 20 144.3
Merchants Academy 19 19 68 58 42 251.6
The Robert Napier School 28 12 68 39 46 292.8
Bishop of Rochester Academy 10 5 70 50 60 298.8
2012
The Rushden Community College 35 26 3 0 54 326.5
Culverhay School 10 19 30 40 20 199.3
Raincliffe School 6 11 50 50 33 211.5
The Coseley School 35 20 60 51 60 262.7
Fleetwood High School 34 22 62 38 24 272.9
John Spendluffe Foundation Technology College 14 12 64 50 43 283.6
Parklands High School 13 18 69 23 8 143.7
Frank F Harrison Engineering College 20 12 70 35 60 188.3
2013
Gloucester Academy 18 13 44 28 50 226.8
Christ the King Catholic and Church of England VA School 22 22 55 32 41 256.5
Aireville School 23 23 61 35 57 267.9
Manchester Creative and Media Academy for Boys 16 19 63 50 50 244.9
Fearns Community Sports College 22 13 64 36 59 306.0
Unity College Blackpool 21 18 67 57 52 277.1
The Mirus Academy 23 13 70 57 52 201.4

.

The size of the high attainer population in these schools varies between 6 (the minimum for which statistics are published) and 35, with an average of just under 20.

The percentage of high attainers within each school’s cohort ranges from 5% to 26% with an average of slightly over 16%.

This compares with a national average in 2013 for all state-funded schools of 32.4%, almost twice the size of the average cohort in this sample. All 20 schools here record a high attainer population significantly below this national average.

This correlation may be significant – tending to support the case that high attainers are more likely to struggle in schools where they are less strongly concentrated – but it does not prove the relationship.

Achievement against the GCSE threshold measure falls as low as 3% (Rushden in 2012) but this was reportedly attributable to the school selecting ineligible English specifications.

Otherwise the poorest result is 30% at Culverhay, also in 2012, followed by Gloucester Academy (44% in 2013) and Raincliffe (50% in 2012). Only these four schools have recorded performance at or below 50%.

Indeed there is a very wide span of performance even amongst these small samples, especially in 2012 when it reaches an amazing 67 percentage points (40 percentage points excluding Rushden). In 2013 there was a span of 26 percentage points and in 2011 a span of 14 percentage points.

The overall average amongst the 20 schools is almost 58%. This varies by year. In 2011 it was 64%, in 2012 it was significantly lower at 51% (but rose to 58% if Rushden is excluded) and in 2013 it was 61%.

This compares with a national average for high attainers in state-funded schools of 94.7% in 2013. The extent to which some of these outlier schools are undershooting the national average is truly eye-watering.

Turning to the progress measures, one might expect even greater variance, given that so many more schools fail to clear this element of the official floor targets with their high attainers.

The overall average across these 20 schools is 41% in English and 44% in maths, suggesting that performance is slightly stronger in maths than English.

But in 2011 the averages were 49% in English and 42% in maths, reversing this general pattern and producing a much wider gap in favour of English.

In 2012 they were 36% in English and 38% in maths, but the English average improves to 41% if Rushden’s result is excluded. This again bucks the overall trend.

The overall average is cemented by the 2013 figures when the average for maths stood at 53% compared with 42% for English.

Hence, over the three years, we can see that the sharp drop in English in 2012 – most probably attributable to the notorious marking issue – was barely recovered in 2013. Conversely, a drop in maths in 2012 was followed by a sharp recovery in 2013.

The small sample size calls into question the significance of these patterns, but they are interesting nevertheless.

The comparable national averages among all state-funded schools in 2013 were 86.2% in English and 87.8% in maths. So the schools in this sample are typically operating at around half the national average levels. This is indeed worse than the comparable record on the threshold measure.

That said, the variation in these results is again huge – 35 percentage points in English (excluding Rushden) and as much as 52 percentage points in maths.

There is no obvious pattern in these schools’ comparative performance in English and maths. Ten schools scored more highly in English and nine in maths, with one school recording equally in both. English was in the ascendancy in 2011 and 2012, but maths supplanted it in 2013.

The final column in Table 2 shows the average point score (APS) for high attainers’ best eight GCSE results. There is once more a very big range, from 144.3 to 326.5 – over 180 points – compared with a 2013 national average for high attainers in state-funded schools of 377.6.

The schools at the bottom of the distribution are almost certainly relying heavily on GCSE-equivalent qualifications, rather than pushing their high attainers towards GCSEs.

Those schools that record relatively high APS alongside relatively low progress scores are most probably taking their high attaining learners with L5 at KS2 to GCSE grade C, but no further.

.

Changes in Performance from 2011 to 2013

Table 3, below, shows how the performance of the 2011 sample changed in 2012 and 2013, while Table 4 shows how the 2012 sample performed in 2013.

The numbers in green show improvements compared with the schools’ 2011 baselines and those in bold are above my illustrative high attainer floor target. The numbers in red are those which are lower than the schools’ 2011 baselines.

.

Table 3: Performance of the 2011 Sample in 2012 and 2013

Name             % HA  5+ A*-C incl E+M      3+ LOP E    3+ LOP M
11 12 13 11 12 13 11 12 13 11 12 13
Carter Community School 13 14 13 56 100 92 56 80 75 44 80 33
Hadden Park High School 13 15 8 60 87 75 40 80 75 20 53 50
Merchants Academy 19 16 20 68 79 96 58 79 88 42 47 71
The Robert Napier School 12 12 11 68 83 96 39 59 92 46 62 80
Bishop of Rochester Academy 5 7 8 70 83 73 50 67 47 60 75 53

.

All but one of the five schools showed little variation in the relative size of their high attainer populations over the three years in question.

More importantly, all five schools made radical improvements in 2012.

Indeed, all five exceeded the 5+ GCSE threshold element of my illustrative floor target in both 2012 and 2013 though, more worryingly, three of the five fell back somewhat in 2013 compared with 2012, which might suggest that short term improvement is not being fully sustained.

Four of the five exceeded the English progress element of the illustrative floor target in 2012 while the fifth – Robert Napier – missed by only 1%.

Four of the five also exceeded the floor in 2013, including Robert Napier which made a 43 percentage point improvement compared with 2012. On this occasion, Bishop of Rochester was the exception, having fallen back even below its 2011 level.

In the maths progress element, all five schools made an improvement in 2012, three of the five exceeding the floor target, the exceptions being Hadden Park and Merchants Academy

But by 2013, only three schools remained above their 2011 baseline and only two – Merchants and Robert Napier – remained above the floor target.

None of the five schools would have remained below my floor target in either 2012 or 2013, by virtue of their improved performance on the 5+ GCSE threshold element, but there was significantly greater insecurity in the progress elements, especially in maths.

There is also evidence of huge swings in performance on the progress measures. Hadden Park improved progression in English by 40 percentage points between 2011 and 2012. Carter Community School almost matched this in maths, improving by 36 percentage points, only to fall back by a huge 47 percentage points in the following year.

Overall this would appear to suggest that this small sample of schools made every effort to improve against the threshold and progress measures in 2012 but, while most were able to sustain improvement – or at least control their decline – on the threshold measure into 2013, this was not always possible with the progress elements.

There is more than a hint of two markedly different trajectories, with one group of schools managing to sustain initial improvements from a very low base and the other group falling back after an initial drive.

Is the same pattern emerging amongst the group of schools that fell below my high attainer floor target in 2012?

.

Table 4: Performance of the 2012 Sample in 2013

Name   % HA  5+ A*-C incl E+M 3+ LOP E 3+ LOP M
12 13  12 13 12 13 12 13
The Rushden Community College 26 23 3 90 0 74 54 87
Culverhay School 19 12 30 67 40 67 20 67
Raincliffe School 11 - 50 - 50 - 33 -
The Coseley School 20 26 60 88 51 82 60 78
Fleetwood High School 22 24 62 84 38 36 24 67
John Spendluffe Foundation Technology College 12 15 64 100 50 61 43 83
Parklands High School 18 11 69 78 23 56 8 56
Frank F Harrison Engineering College 12 13 70 70 35 57 60 52

.

We must rule out Raincliffe, which closed, leaving seven schools under consideration.

Some of these schools experienced slightly more fluctuation in the size of their high attainer populations – and over the shorter period of two years rather than three.

Six of the seven managed significant improvements in the 5+ GCSE threshold with the remaining school – Frank F Harrison – maintaining its 2012 performance.

Two schools – Frank F Harrison and Culverhay did not exceed the illustrative floor on this element.  Meanwhile John Spendluffe achieved a highly creditable perfect score, comfortably exceeding the national average for state-funded schools. Rushden was not too far behind.

There was greater variability with the progress measures. In English, three schools remained below the illustrative floor in 2013 with one – Fleetwood High – falling back compared with its 2012 performance.

Conversely, Coseley improved by 31 percentage points to not far below the national average for state-funded schools.

In maths two schools failed to make it over the floor. Parklands made a 48 percentage point improvement but still fell short, while Frank F Harrison fell back eight percentage points compared with its 2012 performance.

On the other hand, Rushden and John Spendluffe are closing in on national average performance for state-funded schools. Both have made improvements of over 30 percentage points.

Of the seven, only Frank F Harrison would remain below my overall illustrative floor target on the basis of its 2013 performance.

Taking the two samples together, the good news is that many struggling schools are capable of making radical improvements in their performance with high attainers.

But question marks remain over the capacity of some schools to sustain initial  improvements over subsequent years.

 .

What Interventions Have Impacted on these Schools?

Table 5 below reveals how different accountability and school improvement interventions have been brought to bear on this sample of 20 schools since 2011.

.

Table 5: Interventions Impacting on Sample Schools 2011-2014

Name Floor Targets Most recent Inspection Ofsted Rating (Pre-) warning notice Academised
2011
Carter Community School  .FT 2011. FT 2013  .29/11/12. NYI as academy 2 Sponsored
Hadden Park High School  .FT 2011.FT 2012

.FT 2013

 .13/11/13 .NYI as academy SM Sponsored
Merchants Academy  .FT 2011 .FT 2012  .9/6/11 2
The Robert Napier School  .FT 2011.FT 2012  .17/09/09.NYI as academy 3 Sponsored
Bishop of Rochester Academy  .FT 2011.FT 2013  .28/6/13 3 PWN 3/1/12
2012
The Rushden Community College FT 2012  .10/11/10.NYI as academy 3 Sponsored
Culverhay School  .FT 2011 .FT 2012

.(FT 2013)

 .11/1/12 .NYI as academy SM Sponsored
Raincliffe School  .FT 2012  .19/10/10 3 Closed
The Coseley School  .FT 2012  .13/9/12 SM
Fleetwood High School  .FT 2012 .FT 2013  .20/3/13 SWK
John Spendluffe Foundation Technology College  .FT 2012  .3/3/10 .As academy    18/9/13 .1.2 Academy converter 9/11
Parklands High School  .FT 2011.FT 2012

.FT 2013

 .5/12/13 SM Discussing sponsorship
Frank F Harrison Engineering College  .FT 2011.FT 2012

.(FT 2013)

 .5/7/11.See Mirus Academy below 3 Now Mirus Academy (see below)
2013
Gloucester Academy  .FT 2011.FT 2012

 .FT 2013

 .4/10/12 SWK  .PWN 16/9/13.WN 16/12/13
Christ the King RC and CofE VA School  .FT 2011.FT 2012

.FT 2013

 .18/9/12 SM Closed
Aireville School  .FT 2012.FT 2013  .15/5/13 SM
Manchester Creative and Media Academy for Boys  .FT 2011.FT 2012

.FT 2013

 .13/6/13 SWK PWN 3/1/12
Fearns Community Sports College  .FT 2011.FT 2013  .28/6/12 3
Unity College Blackpool .  .FT 2011 .FT 2012

.FT 2013

 .9/11/11.NYI as academy 3 Sponsored
The Mirus Academy  .FT 2013  .7/11/13 SM

 .

Floor Targets

The first and obvious point to note is that every single school in this list fell below the official floor targets in the year in which they also undershot my illustrative high attainers’ targets.

It is extremely reassuring that none of the schools returning particularly poor outcomes with high attainers are deemed acceptable performers in generic terms. I had feared that a few schools at least would achieve this feat.

In fact, three-quarters of these schools have fallen below the floor targets in at least two of the three years in question, while eight have done so in all three years, two having changed their status by becoming academies in the final year (which, strictly speaking, prevents them from scoring the hat-trick). One has since closed.

Some schools appear to have been spared intervention by receiving a relatively positive Ofsted inspection grade despite their floor target records. For example, Carter Community School had a ‘good’ rating sandwiched between two floor target appearances, while Merchants Academy presumably received its good rating before subsequently dropping below the floor.

John Spendluffe managed an outstanding rating two years before it dropped below the floor target and was rated good – in its new guise as an academy – a year afterwards.

The consequences of falling below the floor targets are surprisingly unclear, as indeed are the complex rules governing the wider business of intervention in underperforming schools.

DfE press notices typically say something like:

Schools below the floor and with a history of underperformance face being taken over by a sponsor with a track record of improving weak schools.’

But of course that can only apply to schools that are not already academies.

Moreover, LA-maintained schools may appeal to Ofsted against standards and performance warning notices issued by their local authorities; and schools and LAs may also challenge forced academisation in the courts, arguing that they have sufficient capacity to drive improvement.

As far as I can establish, it is nowhere clearly explained what exactly constitutes a ‘history of underperformance’, so there is inevitably a degree of subjectivity in the application of this criterion.

Advice elsewhere suggests that a school’s inspection outcomes and ‘the local authority’s position in terms of securing improvement as a maintained school’ should also be taken into account alongside achievement against the floor targets.

We do not know what weighting is given to these different sources of evidence, nor can we rule out the possibility that other factors – tangible or intangible – are also weighed in the balance.

Some might argue that this gives politicians the necessary flexibility to decide each case on its merits, taking careful account of the unique circumstances that apply rather than imposing a standard set of cookie-cutter judgements.

Others might counter that the absence of standard criteria, imposed rigorously but with flexibility to take additional special circumstances in to account, lays such decisions unnecessarily open to dispute and is likely to generate costly and time-consuming legal challenge

.

Academy Warning Notices

When it comes to academies:

‘In cases of sustained poor academic performance at an academy, ministers may issue a pre-warning notice to the relevant trust, demanding urgent action to bring about substantial improvements, or they will receive a warning notice. If improvement does not follow after that, further action – which could ultimately lead to a change of sponsor – can be taken. In cases where there are concerns about the performance of a number of a trust’s schools, the trust has been stopped from taking on new projects.’

‘Sustained poor academic performance’ may or may not be different from a ‘history of underperformance’ and it too escapes definition.

One cannot but conclude that it would be very helpful indeed to have some authoritative guidance, so that there is much greater transparency in the processes through which these various provisions are being applied, to academies and LA-maintained schools alike.

In the absence of such guidance, it seems rather surprising that only three of the academies in this sample – Bishop of Rochester, Gloucester and Manchester Creative and Media – have received pre-warning letters to date, while only Gloucester’s has been superseded by a full-blown warning notice. None of these mention specifically the underperformance of high attainers.

  • Bishop of Rochester received its notice in January 2012, but subsequently fell below the floor targets in both 2012 and 2013 and – betweentimes – received an Ofsted inspection rating of 3 (‘requires improvement’).
  • Manchester Creative and Media also received its pre-warning notice in January 2012. It too has been below the floor targets in both 2012 and 2013 and was deemed to have serious weaknesses in a June 2013 inspection.
  • Gloucester received its pre-warning notice much more recently, in September 2013, followed by a full warning notice just three months later.

These pre-warning letters invite the relevant Trusts to set out within 15 days what action they will take to improve matters, whereas the warning notices demand a series of specific improvements with a tight deadline. (In the case of Gloucester Academy the notice issued on 16 December 2013 imposing a deadline of 15 January 2014. We do not yet know the outcome.)

Other schools in my sample have presumably been spared a pre-warning letter because of their relatively recent acquisition of academy status, although several other 2012 openers have already received them. One anticipates that more will attract such attention in due course.

 .

Ofsted Inspection

The relevant columns of Table 5 reveal that, of the 12 schools that are now academies (taking care to count Harrison/Mirus as one rather than two), half have not yet been inspected in their new guise.

As noted above, it is strictly the case that, when schools become academies – whether sponsored or via conversion – they are formally closed and replaced by successor schools, so the old inspection reports no longer apply to the new school.

However, this does not prevent many academies from referring to such reports on their websites – and they do have a certain currency when one wishes to see whether or not a recently converted academy has been making progress.

But, if we accept the orthodox position, there are only six academies with bona fide inspection reports: Merchants, Bishop of Rochester, John Spendluffe, Gloucester, Manchester Creative and Media and Mirus.

All five of the LA-maintained schools still open have been inspected fairly recently: Coseley, Fleetwood, Parklands, Aireville and Fearns.

This gives us a sample of 11 schools with valid inspection reports:

  • Two academies are rated ‘good’ (2)  – Merchants and John Spendluffe;
  • One academy – Bishop of Rochester – and one LA-maintained school –  Fearns – ‘require improvement’ (3);
  • Two academies – Gloucester and Manchester – and one LA-maintained school – Fleetwood – are inadequate (4) having serious weaknesses and
  • One academy – Mirus – and three LA-maintained schools – Parklands, Coseley and Aireville – are inadequate (4) and in Special Measures.

The School Inspection Handbook explains the distinction between these two  variants of ‘inadequate’:

‘A school is judged to require significant improvement where it has serious weaknesses because one or more of the key areas is ‘inadequate’ (grade 4) and/or there are important weaknesses in the provision for pupils’ spiritual, moral, social and cultural development. However, leaders, managers and governors have been assessed as having the capacity to secure improvement

…A school requires special measures if:

  • it is failing to give its pupils an acceptable standard of education and
  • the persons responsible for leading, managing or governing are not demonstrating the capacity to secure the necessary improvement in the school.’

Schools in each of these categories are subject to more frequent monitoring reports. Those with serious weaknesses are typically re-inspected within 18 months, while, for those in special measures, the timing of re-inspection depends on the school’s rate of improvement.

It may be a surprise to some that only seven of the 11 are currently deemed inadequate given the weight of evidence stacked against them.

There is some support for the contention that Ofsted inspection ratings, floor target assessments and pre-warning notices do not always link together as seamlessly as one might imagine, although apparent inconsistencies may sometimes arise from the chronological sequence of these different judgements.

But what do these 11 reports say, if anything, about the performance of high attainers? Is there substantive evidence of a stronger focus on ‘the most able’ in those reports that have issued since September 2013?

.

The Content of Ofsted Inspection Reports

Table 6, below, sets out what each report contains on this topic, presenting the schools in the order of their most recent inspection.

One might therefore expect the judgements to be more specific and explicit in the three reports at the foot of the table, which should reflect the new guidance introduced last September. I discussed that guidance at length in this October 2013 post.

.

Table 6: Specific references to high attainers/more able/most able in inspection reports

Name Date Outcome Comments
Merchants Academy 29/6/11 Good (2) In Year 9… an impressive proportion of higher-attaining students…have been entered early for the GCSE examinations in mathematics and science. Given their exceptionally low starting points on entry into the academy, this indicates that these students are making outstanding progress in their learning and their achievement is exceptional.More-able students are fast-tracked to early GCSE entry and prepared well to follow the InternationalBaccalaureate route.
Fearns Community Sports College 28/6/12 Requires improvement (3) Setting has been introduced across all year groups to ensure that students are appropriately challenged and supported, especially more-able students. This is now beginning to increase the number of students achieving higher levels earlier in Key Stage 3.
The Coseley School 13/9/12 Special Measures (4) Teaching is inadequate because it does not always extend students, particularly the more able.What does the school need to do to improve further?Raise achievement, particularly for the most able, by ensuring that:

  • work consistently challenges and engages all students so that they make good progress in lessons
  • challenging targets are set as a minimum expectation
  • students do not end studies in English language and mathematics early without having the chance to achieve the best possible grade
  • GCSE results in all subjects are at least in line with national expectations.

Target setting is not challenging enough for all ability groups, particularly for the more-able students who do not make sufficient progress by the end of Key Stage 4.

Gloucester Academy 4/10/12 Serious Weaknesses (4) No specific reference
Fleetwood High School 20/3/13 Serious Weaknesses(4) No specific reference
Aireville School 15/5/13 Special Measures(4) Teachers tend to give the same task to all students despite a wide range of ability within the class. Consequently, many students will complete their work and wait politely until the teacher has ensured the weaker students complete at least part of the task. This limits the achievement of the more-able students and undermines the confidence of the least-able.There is now a good range of subjects and qualifications that meet the diverse needs and aspirations of the students, particularly the more-able students.
Manchester Creative and Media Academy for Boys 13/6/13 Serious Weaknesses(4) The most-able boys are not consistently challenged to attain at the highest levels. In some lessons they work independently and make rapid progress, whereas on other occasions their work is undemanding.What does the academy need to do to improve further?Improve the quality of teaching in Key Stages 3 and 4 so that it is at least good leading to rapid progress and raised attainment for all groups of boys, especially in English, mathematics and science by…  ensuring that tasks are engaging and challenge all students, including the most-able.The most-able boys receive insufficient challenge to enable them to excel. Too many lessons donot require them to solve problems or link their learning to real-life contexts.In some lessons teachers’ planning indicates that they intend different students to achieve different outcomes, but they provide them all with the same tasks and do not adjust the pace or nature of work for higher- or lower-attaining students. This results in a slow pace of learning and some boys becoming frustrated.
Bishop of Rochester Academy 28/6/13 Requires improvement (3) No specific reference
John Spendluffe Foundation Technology College 18/9/13 Good (2) Not enough lessons are outstanding in providing a strong pace, challenge and opportunities for independent learning, particularly for the most able.The 2013 results show a leap forward in attainment and progress, although the most able could still make better progress.Leadership and management are not outstanding because the achievement of pupils, though improving quickly, has not been maintained at a high level over a period of time, and a small number of more-able students are still not achieving their full potential.
The Mirus Academy 7/11/13 Special Measures (4) The academy’s early entry policy for GCSE has made no discernible difference to pupils’ achievement, including that of more able pupils.
Parklands High School 5/12/13 Special Measures (4) The achievement of students supported by the pupil premium generally lags behind that of their classmates. All groups, including themost able students and those who have special educational needs, achieve poorly.Students who join the school having achieved Level 5 in national Key Stage 2 tests in primary school fare less well than middle attainers, in part due to early GCSE entry. They did a little better in 2013 than in 2012.

.

There is inconsistency within both parts of the sample – the first eight reports that pre-date the new guidance and the three produced subsequently.

Three of the eleven reports make no specific reference to high attainers/most able learners, all of them undertaken before the new guidance came into effect.

In three more cases the references are confined to early entry or setting, one of those published since September 2013.

Only four of the eleven make what I judge to be substantive comments:

  • The Coseley School (special measures) – where the needs of the most able are explicitly marked out as an area requiring improvement;
  • The Manchester Creative and Media Academy for Boys (serious weaknesses) – where attention is paid to the most able throughout the report;
  • John Spendluffe Foundation Technology College (good) – which includes some commentary on the performance of the most able; and
  • Parklands High School (special measures) – which also provides little more than the essential minimum coverage.

The first two predate the new emphasis on the most able, but they are comfortably the most thorough. It is worrying that not all reports published since September are taking the needs of the most able as seriously as they might.

One might expect that, unconsciously or otherwise, inspectors are less ready to single out the performance of the most able when a school is inadequate across the board, but the small sample above does not support this hypothesis. Some of the most substantive comments relate to inadequate schools.

It therefore seems more likely that the variance is attributable to the differing capacity of inspection teams to respond to the new emphases in their inspection guidance. This would support the case made in my previous post for inspectors to receive additional guidance on how they should interpret the new requirement.

.

Conclusion

This post established an illustrative floor target to identify a small sample of 20 schools that have demonstrated particularly poor performance with high attainers in the Performance Tables for 2011, 2012 or 2013.

It:

  • Compared the performance of these schools in the year in which they fell below the floor, noting significant variance by year and between institutions, but also highlighting the fact that the proportion of high attainers attending these schools is significantly lower than the national average for state-funded schools.
  • Examined the subsequent performance of schools below the illustrative floor in 2011 and 2012, finding that almost all made significant improvements in the year immediately following, but that some of the 2011 cohort experienced difficulty in sustaining this improvement across all elements into a second year. It seems that progress in English, maths or both are more vulnerable to slippage than the 5+ A*-C GCSE threshold measure.
  • Confirmed – most reassuringly – that every school in the sample fell below the official, generic floor targets in the year in which they also undershot my illustrative high attainer floor targets.
  • Reviewed the combination of assessments and interventions applied to the sample of schools since 2011, specifically the interaction between academisation, floor targets, Ofsted inspection and (pre)warning notices for academies. These do not always point in the same direction, although chronology can be an extenuating factor. New guidance about how these and other provisions apply and interact would radically improve transparency in a complex and politically charged field.
  • Analysed the coverage of high attainers/most able students in recent inspection reports on 11 schools from amongst the sample of 20, including three published after September 2013 when new emphasis on the most able came into effect. This exposed grave inconsistency in the scope and quality of the coverage, both before and after September 2013, which did not correlate with the grade of the inspection. Inspectors would benefit from succinct additional guidance.

In the process of determining which schools fell below my high attainers floor target, I also identified the schools that undershot one or other of the elements but not both. This wider group included 46 schools in 2011, 52 schools in 2012 and 34 schools in 2013.

Several of these schools reappear in two or more of the three years, either in their existing form or following conversion to academy status.

Together they constitute a ‘watch list’ of more than 100 institutions, the substantial majority of which remain vulnerable to continued underperformance with their high attainers for the duration of the current accountability regime.

The chances are that many will also continue to struggle following the introduction of the new ‘progress 8’ floor measure from 2015.

Perhaps unsurprisingly, the significant majority are now sponsored academies.

I plan to monitor their progress.

.

*Apologies for this rather tabloid title!

.

GP

February 2014

The 2013 Transition Matrices and High Attainers’ Performance

.

Data Overload courtesy of opensourceway

Data Overload courtesy of opensourceway

Since last year’s post on the Secondary Transition Matrices attracted considerable interest, I thought I’d provide a short commentary on what the 2013 Matrices – primary and secondary – tell us about the national performance of high attainers.

This note is a postscript to my recent offerings on:

and completes the set of benchmarking resources that I planned to make available.

I am using the national matrices rather than the interactive matrices (which, at the time of writing, are not yet available for 2013 results). I have included a few figures from the 2012 national matrices for comparative purposes.

According to Raise Online, the national matrices are derived from the data for ‘maintained mainstream and maintained and non-maintained special schools’.

They utilise KS2 fine points scores as set out below.

Sub Level Points Fine points range
6 39 36-41.99
5A 34-35.99
5B 33 32-33.99
5C 30-31.99
4A 28-29.99
4B 27 26-27.99
4C 24-25.99
3A 22-23.99
3B 21 20-21.99
3C 18-19.99

.

2013 Primary Transition Matrices

The Primary Matrices track back the KS1 performance of learners completing KS2 tests in 2013.

.

Reading

.

2013 primary reading TM.

This table shows that:

  • 12% of KS1 learners with L4 reading secured the minimum expected 2 levels of progress by securing L6 at the end of KS2. It is not possible for such learners to make more than 2 levels of progress. Almost all the remaining 88% of Level 4 learners made a single level of progress to Level 5.
  • By comparison, just 1% of learners achieving Level 3 in KS1 made 3 levels of progress to Level 6 (the same percentage as in 2012).
  • 87% of KS1 learners achieving L3 in reading secured the expected 2 or more levels of progress, 85% of them making 2 levels of progress to L5. However, some 13% made only 1 level of progress to L4. (In 2012, 89% of those with L3 at secured L5 and 10% reached L4.)
  • The proportion of learners with L3 in reading at KS1 who made the expected 2 levels of progress was lower than the proportions of learners with L2 overall, L2A, or L2B doing so. The proportion exceeding 2 levels of progress was far higher for every other level of KS1 achievement. (This was also true in 2012.)

.

Writing

.

2013 primary writing TM.

This table shows that:

  • 61% of learners achieving L4 in writing at KS1 made the requisite 2 levels of progress to L6 at KS2. Such learners are unable to make more than 2 levels of progress. The remaining 39% of L4 learners made a single level of progress to L5.
  • This compares with 9% of learners with L3 at KS1 who made 3 levels of progress to L6 (up from 6% in 2012). A further 2% of learners with L2A made 4 levels of progress to L6.
  • 89% of learners with L3 in KS1 writing made the expected 2 or more levels of progress, 80% of them making 2 levels of progress to L5. But 11% made only a single level of progress to L4. (In 2012, 79% of those with L3 at KS1 reached L5 and 15% made only L4.)
  • The proportion of learners with L3 at KS1 in writing achieving the expected 2 levels of progress was lower than the proportions of learners with L2 overall, L2A or L2B, or even L1 doing so. The proportion exceeding 2 levels of progress was far higher for every other level of KS1 achievement with the exception of L2C. (A similar pattern was evident in 2012.)

.

Maths

.

2013 primary maths TM.

This table shows that:

  • 89% of those achieving L4 in maths at KS1 made the requisite 2 levels of progress to L6 in KS2. These learners are unable to make more than 2 levels of progress. But the remaining 11% of those with L4 at KS1 made only a single level of progress to KS2 L5.
  • This compares with 26% of learners at L3 in KS1 maths who made 3 levels of progress to KS2 L6 (up significantly from 14% in 2012). In addition, 4% of those at KS1 L2A and 1% of those at 2B also managed 4 levels of progress to KS2 L6.
  • 90% of learners with L3 in KS1 maths made the expected 2 or more levels of progress to L5, 64% making 2 levels of progress to L5. But a further 10% made only a single level of progress to KS2 L4. (In 2012, 74% of those with L3 at KS1 made it to KS2 L5 and 11% secured L4.)
  • The proportion of learners with L3 at KS1 in maths who achieved the expected 2 levels of progress was lower than the proportions of those with KS1 L2A or L2B doing so. The proportion of learners exceeding 2 levels of progress was significantly higher for those with KS1 L2 overall, those with L2A, and even those with L1, but it was lower for those with L2B and especially L2C. (In 2012 the pattern was similar, but the gap between the proportions with L2B and L3 exceeding 2 levels of progress has narrowed significantly.)

.

Key Challenges

The key challenges in respect of high attainers in the primary sector are to:

  • Enable a higher proportion of learners with L4 at KS1 to make the expected 2 levels of progress to KS2 L6. There is a particular problem in reading where 88% of these learners are making a single level of progress.
  • Enable a higher proportion of learners with L3 at KS1 to make 3 levels of progress to KS2 L6. Reading is again the least advanced, but there is huge scope for improvement across the board. Efforts should be made to close the gaps between L2A and L3 making three levels of progress, which currently stand at 55 percentage points (reading), 49 percentage points (writing) and 30 percentage points (maths). For the duration of their existence, increasing take-up of KS2 L6 tests should secure further improvement.
  • Increase the proportions of learners with L3 at KS1 making 2 levels of progress so they are comparable with what is achieved by those with L2A and L2B at KS1. There are currently gaps of 11 percentage points (reading), 10 percentage points (writing) and 9 percentage points (maths) between those with L3 and those with L2A. The gaps between those with L3 and those with L2B are 5 percentage points (reading), 8 percentage points (writing) and 1 percentage point (maths).
  • Ensure that far fewer learners with L3 at KS1 manage only a single level of progress across KS2. The current levels – 13% in reading, 11% in writing and 10% in maths – are unacceptable.

.

Secondary

The Secondary Matrices track back the KS2 performance of learners completing GCSEs in 2013.

.

English

.

2013 secondary Englsih sublevels TM.

The table shows:

  • 97% of KS2 learners achieving 5A in English secured at least 3 levels of progress from KS2 to KS4 in 2013. This compares with 92% of learners achieving 5B and 74% of learners achieving 5C. (The comparable figures in 2012 were 98%, 92% and 70% respectively.)
  • 89% of KS2 learners achieving 5A in English achieved 4 or more levels of progress from KS2 to KS4 in 2013, so achieving an A* or A grade at GCSE, compared with 66% of those achieving 5B and 33% of those achieving 5C. (The comparable figures in 2012 were 87%, 64% and 29% respectively.)
  • The percentages of learners with 4A in English at KS2 who completed 3 and 4 or more levels of progress – 87% and 46% respectively – were significantly higher than the comparable percentages for learners achieving 5C.
  • 53% of KS2 learners achieving 5A in English made 5 levels of progress by achieving A* at GCSE, compared with 23% of those achieving 5B and 6% of those achieving 5C. (These are significantly higher than the comparable figures for 2012, which were 47%, 20% and 4% respectively).
  • 1% of KS2 learners achieving 5A at KS2 made only two levels of progress to GCSE grade C, compared with 6% of those with 5B and 22% of those with 5C. (These percentages have fallen significantly compared with 2012, when they were 3%, 13% and 30% respectively.)

.

Maths

.2013 secondary maths sublevels TM.

This table shows:

  • 97% of those achieving 5A in maths secured at least 3 levels of progress from KS2 to KS4, whereas 88% of learners achieving 5B did so and 70% of learners at 5C. (The comparable 2012 figures were 96%, 86% and 67% respectively.)
  • 85% of KS2 learners achieving 5A in maths made 4 or more levels of progress in 2013 to GCSE A* or A grades, compared with 59% of those at 5B and 31% of those at 5C. (The comparable 2012 figures were 84%, 57% and 30%.)
  • The percentage of learners achieving 4A in maths at KS2 who completed 3 and 4 or more levels of progress – 91% and 43% respectively – were significantly higher than the percentages of those with 5C who did so.
  • 53% of KS2 learners with 5A in maths made 5 levels of progress to achieve an A* grade in maths, compared with 22% of those with 5B and 6% of those with 5C. (The comparable figures for 2012 were 50%, 20% and 6% respectively).
  • 3% of learners with 5A at KS2 made only two levels of progress to GCSE grade C, compared with 11% of those with 5B and 27% of those with 5C. (These percentages were 3%, 13% and 30% in 2012.)

.

Key challenges

The key challenges in respect of high attainers in the secondary sector are to:

  • Ensure that, so far as possible, all learners with L5 at KS2 make at least 3 levels of progress to at least GCSE grade B. Currently more than 1 in 5 students with Level 5C fail to achieve this in English and more than 1 in 4 fail to do so in maths. Moreover, more than 1 in 10 of those with 5B at KS2 fall short of 3 levels of progress in maths. This is disappointing.
  • Ensure that a higher proportion of learners with L5 at KS2 make 4 and 5 levels of progress. The default expectation for those with L5A at KS2 should be an A* Grade at GCSE (5 levels of progress) while the default for those with L5B at KS2 should be at least Grade A at GCSE (4 levels of progress). Currently 47% of those with L5A are falling short of A* in both English and maths, while 34% of those with L5B are falling short of A*/A in English while 41% are doing so in maths.
  • Narrow the gaps between the performance of those with L5C at KS2 and those with L4A. Currently there are 13 percentage point gaps between the proportions making the expected 3 levels of progress and between the proportions exceeding 3 levels of progress in English, while in maths there are gaps of 21 percentage points between those making 3 levels of progress and of 12 percentage points between those exceeding 3 levels of progress.

.

GP

January 2014

High Attainment in the 2013 Secondary and 16-18 Performance Tables

.

.

This post reviews high attainment and high attaining student data in the 2013 Secondary and 16-18 Performance Tables relating

Data Overload courtesy of opensourceway

to GCSE and A level respectively. It compares key outcomes with those reported in last year’s tables.

It also draws extensively on two accompanying statistical publications:

and derives three year trends, from these and the comparable 2011 and 2012 publications, focused primarily on variations by sector and school admission arrangements.

This post complements a briefer analysis of High Attainment in the 2013 Primary School Performance Tables published on 12 December 2013 and updates last year’s High Attaining Students in the 2012 Secondary School Performance Tables (January 2013).

This year’s secondary/post-16 analysis is presented in a somewhat different format, organised into sections relating to key measures, beginning with GCSE and moving on to A level.

A few preliminaries:

There are sometimes discrepancies between the figures given in the Tables and those in the supporting statistical publications that I cannot explain.

The commentary highlights results – some extraordinarily good, others correspondingly poor – from specific institutions identified in the Tables. This adds some richness and colour to what might otherwise have been a rather dry post.

But there may of course be extenuating circumstances to justify particularly poor results which are not allowed for in the Tables. Equally, strong results may not always be solely attributable to the quality of education provided in the institution that secures them.

As always, I apologise in advance for any transcription errors and urge you to report them through the comments facility provided.

Those who prefer not to read the full post will find a headline summary immediately below. The main text provides additional detail but is intended primarily for reference purposes.

.

Headlines

.

Media Coverage

There has been relatively little media coverage of what the Performance Tables reveal about the achievement of high attainers, though one article appeared in the Daily Telegraph.

It said that the high attainer population comprised some 175,800 students of which:

  • about 9,300 ‘failed to gain five good GCSEs at grades A* to C, including English and maths’;
  • around 48% per cent (over 84,200) did not pass the EBacc and 35% [approaching 62,000] did not enter all the necessary subjects;
  • in English almost 14% [24,000] ‘effectively went backwards in English by gaining lower scores at GCSE level than comparable tests taken at 11, while 12% [21,000] did so in maths’.

The figures in square brackets are my own, derived from the percentages provided in the article.

The final point suggests that sizeable minorities of high attainers achieved the equivalent of Level 4 in English and maths GCSEs, but this is incorrect.

These figures relate to the proportion of high attainers who did not make at least three levels of progress from KS2 to KS4 in English and maths (see below) – quite a different matter.

.

Headlines from this analysis

The following extended bullet points summarise the key findings from my own analysis:

  • The high attainer population constitutes almost exactly one third of the population of mainstream state-funded schools. A gender gap that had almost closed in 2012 has widened again in favour of girls. There are significant variations between school types – for example just over 20% of students attending sponsored academies are high attainers compared with just under 40% in academy converters. The population in free schools, UTCs and studio schools has fallen by 11.5% since 2012, presumably as a consequence of the sector’s rapid expansion. Only 90% of the selective school cohort constitutes high attainers, which suggests 10% of their intake are middle attainers who perform well on ability-based 11+ assessments. The selective school high attainer population has fallen by 1.4% since 2011. Ten selective schools record that their cohort consists entirely of high attainers, but some selective schools register a cohort in which two-thirds or fewer students are high attainers. This is comfortably lower than some non-selective schools, raising awkward questions about the nature of selective education. Although there are no schools with no high attainers, two schools recorded 3% and 99 have fewer than 10% (down from 110 in 2012). Academies in coastal towns are well-represented. The schools containing these small high attaining groups demonstrate huge variations in high attainer performance. This warrants further investigation.
  • 60.6% of all students in state-funded schools achieved 5 or more GCSEs at grades A*-C (or equivalent) including GCSEs in English and maths, a 1.8% improvement compared with 2012. But the success rate for high attainers improved by only 0.7% to 94.7%. This is better than the 0.2% fall in the success rate amongst low attainers but falls well short of the 2. 3% improvement for middle attainers. One in every twenty high attainers continues to miss this essential benchmark. But 50 more schools recorded 100% returns than in 2012 – and 19 fewer schools were at 75% or below. Apart from selective schools falling foul of the ineligibility of some IGCSEs, Ark King’s Academy in Birmingham was the lowest performer at 33%. Trends vary considerably according to school type. Free schools, UTCs and studio schools have improved by 4.2% since 2012, which must be partly a consequence of the growth of that sector. Meanwhile high attainers in selective schools have fallen back by 2.0% (and selective schools overall by 2.3%) since 2011. It is unlikely that idiosyncratic IGCSE choices are solely responsible. The profiles of sponsored and converter academies are still markedly different, though the gap between their high attainers’ performance has halved since 2011, from 5.3 percentage points to 2.7 percentage points.
  • There were big increases in the percentage of all students entered for all EBacc subjects in state-funded schools – up 12.4% to 35.5% – and the percentage successful – up 6.6% to 22.8%. The comparable entry and success rates for high attainers were 65.0% and 52.1% respectively. The entry rate for 2012 was 46.3%, so that has improved by almost 19 percentage points, a much faster rate of improvement than the headline figure. The success rate has improved from 38.5% last year, so by 13.6 percentage points, more than double the improvement in the headline figure. The EBacc is clearly catching on for high attainers following a relatively slow start. That said, one could make a case that the high attainer success rate in particular remains rather disappointing, since something like one in five high attainers entered for the EBacc fail to convert entry into achievement. Forty-seven schools entered all of their high attainers but only four recorded 100% success, two selective (Chelmsford County High for Girls and Queen Elizabeth’s Barnet) and two comprehensive (St Ursula’s Convent School in Greenwich and The Urswick School in Hackney). Only 55 schools entered no high attainers for the EBacc, compared with 186 in 2012. Seventy-nine schools recorded 0% of high attainers achieving the EBacc, also down significantly, from 235 in 2012. Seven of these were grammar schools, presumably all falling foul of IGCSE restrictions.
  • 70.4% of all students in state-funded schools made at least the expected three levels of progress in English and 70.7% did so in maths. These constitute improvements of 2.4% and 2.0% respectively. High attainers registered 86.2% success in English and 87.8% in maths. Their rates of improvement were broadly comparable with the headline figures, though slightly stronger in English. It remains disturbing that one in seven high attainers fail to make the expected progress in English and 1 in 8 fail to do so in maths. More schools achieved 100% success amongst their high attainers on each measure than in 2012 – 108 in English and 120 in maths. Forty-four schools were at or below 50% on this measure in English, some IGCSE-favouring grammar schools amongst them. Apart from those, the worst performer was Gloucester Academy at 28%. In maths 31 schools were at or below this 50% benchmark and the worst performer was Stafford Sports College at 29%. Six schools managed 50% or below in both English and maths, several of them academies. Amongst those at 50% or below in English, 11 had better rates of performance for both their middle and their low attainers than for their high attainers. Amongst those at 50% or below in maths, only one school achieved this feat – St Peter’s Catholic College of Maths and Computing (!) in Redcar and Cleveland. It is a cause for concern that high attainers in English attending selective schools continue to fall back on this measure and that one in five high attainers in sponsored academies, free schools, UTCs and studios is failing to make three levels of progress in English, while the same is true of maths in sponsored academies.
  • 7.5% of students in state-funded schools and colleges achieved grades of AAB or higher at A level with all three in facilitating subjects, an improvement of 0.1% compared with 2012. But the comparable percentage for students who achieved these grades with at least two in facilitating subjects shot up to 12.1%, an improvement of 4.3% on 2012. There are big variations between sectors, with the percentage achieving the former measure ranging from 3.5% (FE colleges) to 10.4% (converter academies. The figure for selective schools is 21.1%. Turning to the latter measure, percentages vary from 5.4% in mainstream sponsored academies to 16.4% in mainstream converter academies, while selective schools stand at 32.4%. Across all sectors, more students achieve grades AAA or higher in any A level subjects than achieve AAB or higher in three facilitating subjects. The proportion of students achieving AAA or higher in any A levels is falling in most sectors and institutional types, except in free schools, UTCs and studios and in FE colleges. The proportion achieving AAB or higher in any subjects is falling except in sponsored academies and FE colleges. Conversely there are improvements for AAB or higher with all three in facilitating subjects in LA-maintained mainstream schools, sponsored academies, sixth form colleges and FE colleges (and also across all comprehensive schools).  Across all state-funded mainstream schools, the percentage of A level A* grades has fallen back by 0.5% since 2011 while the percentage of A*/A grades has declined by 0.1%.

The full commentary below names 22 schools which perform particularly badly on one or more GCSE high attainer measures (leaving aside selective schools that have adopted ineligible GCSEs).

Of those 22, only nine are below the floor targets and, of those nine, only four are not already academies. Hence the floor targets regime leaves the vast majority of these schools untouched.

The only hope is that these schools will be caught by Ofsted’s renewed emphasis on the attainment and progress of the ‘most able’ learners (though that provision could do with further clarification as this previous post explained).

 

Definitions

The analysis of GCSE performance is focused primarily on high attainers, while the A level analysis is confined to high attainment.

This is a consequence of the way the two sets of performance tables are constructed (such distinctions were brought out more fully in this October 2013 post.)

There is no coverage of A*/A performance at GCSE within the Secondary Tables so we must necessarily rely on performance against standard measures, such as 5+ GCSEs at A*-C including English and maths and the English Baccalaureate (EBacc).

The Government response to the consultation on secondary accountability reform suggests that this will remain the case, with material about achievement of top GCSE grades confined to the supporting Data Portal. It remains to be seen whether this arrangement will give high attainment the prominence it needs and deserves.

The current definition of high attainers is based on prior performance at the end of KS2. Most learners will have taken these KS2 tests five years previously, in 2008:

  • High attainers are those who achieved above Level 4 in KS2 tests – ie their average point score (APS) in English, maths and science tests was 30 or higher.
  • Middle attainers are those who achieved at the expected Level 4 in KS2 tests – ie their APS in these tests was between 24 and 29.99 – and
  • Low attainers are those who achieved below Level 4 in KS2 tests – ie their APS in these tests was under 24.

Since high attainers are determined on the basis of APS across three subjects, the definition will include all-rounders who achieve good (if not outstanding) results across all three tests, as well as some with a relatively spiky achievement profile who compensate for middling performance in one area through very high attainment in another.

Conversely, learners who are exceptionally strong in one subject but relatively poor in the other two are unlikely to pass the APS 30 threshold.

Both the Secondary Tables and the associated statistical publications remain bereft of data about the performance of high attainers from disadvantaged backgrounds and how that compares with the performance of their more advantaged high attaining peers.

This is unfortunate, since schools that are bucking the trend in this respect – achieving a negligible ‘excellence gap’ between their high attainers from advantaged and disadvantaged backgrounds – richly deserve to be celebrated and emulated.

At A level a variety of high attainment measures are reported in the statistical publications, but the Performance Tables focus on the achievement of AAB+ grades in the so-called ‘facilitating subjects’.

The Statement of Intent for last year’s Tables confirmed the intention to introduce:

‘Percentages of students achieving three A levels at grades AAB or higher in facilitating subjects, reflecting the subjects and grades most commonly required by Russell Group and other top universities.’

These subjects are listed as ‘biology, chemistry, physics, mathematics, geography, history, English literature, modern and classical languages.’

Such measures have been widely criticised for their narrowness, the Russell Group itself asserting that:

‘It would be wrong to use this simple indicator as a measure of the number of pupils in a school who are qualified to apply successfully to a Russell Group university.’

Nevertheless, they support one of the Government’s preferred Social Mobility Indicators which compares the percentage of students attending state and independent schools who achieve this measure. (In 2012 the gap was 15.1%, a full percentage point smaller than in 2011.)

There is nothing in the 16-18 Tables about high attainers, although the consultation document on 16-19 accountability reform includes a commitment to:

‘Consider how we can report the results of low, middle and high attainers similarly [to KS4] in the expanded 16-19 performance tables’.

At the time of writing, the response to this consultation has not been published.

.

GCSE Achievement

 .

The High Attainer Population

Before examining the performance data it is important to review the size of the high attaining population and how this varies between genders, sectors and types of school.

Tables 1A, B and C below show that the population has remained relatively stable since 2011. It accounts consistently for almost exactly one third of students in state-funded mainstream schools.

The gender gap amongst high attainers has changed slightly since 2011. The percentage of high attaining girls has fallen back, slightly but consistently, while the percentage of high attaining boys increased in 2012, only to fall back again in 2013.

A gender gap that had almost been eliminated in 2012 has now widened again to a full percentage point. The percentages of high attaining learners of both genders are the lowest they have been over the three year period.

There are significant variations according to sector and school type, since the high attainer population in converter academies is almost double that in sponsored academies, where it constitutes barely a fifth of the student body. This is a strikingly similar proportion to that found in modern schools.

The percentage of high attainers in comprehensive schools is only very slightly lower than the overall figure.

At the other end of the spectrum, the high attaining cohort constitutes around 90% of the selective school population, which begs interesting questions about the nature of the other 10% and possible discrepancies between KS2 results and ability-focused 11+ assessment.

It cannot be the case that the majority of the missing 10% attended independent preparatory schools and did not take KS2 tests, since those without test results are excluded from the calculations.

The underlying trend is downward in all types of school. There has been a huge 11.5% fall in the proportion of high attainers in free schools, UTCs and studio schools. This is presumably consequent upon the expansion of that sector and brings it much more into line with the figures for all maintained mainstream and other comprehensive schools.

Otherwise the most substantial reduction has been in converter academies. The percentage in selective schools has fallen by 1.4% since 2011, twice the rate of decline in comprehensive schools.

.

Table 1A: Percentage of high attainers by sector 2011, 2012 and 2013

All maintained mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools
2013 32.8 30.6 20.5 39.3 31.4
2012 33.6 32.0 20.9 42.5 42.9
2011 33.5 - 20.6 47.5 -

.

Table 1B: Percentage of high attainers by admissions practice 2011, 2012 and 2013

Selective Comprehensive Modern
2013 88.9 30.9 20.5
2012 89.8 31.7 20.9
2011 90.3 31.6 20.4

.

Table 1C: Percentage of high attainers by gender, all state-funded mainstream schools 2011, 2012, 2013

  Boys Girls
2013 32.3 33.3
2012 33.4 33.8
2011 32.6 34.4

 

The 2013 Performance Tables list 10 schools where 100% of pupils are deemed high attainers, all of which are selective. Thirteen selective schools were in this position in 2012.

But there is also a fairly wide spread amongst selective schools, with some recording as few as 70% high attainers, broadly comparable with some prominent non-selective schools.

For example, Dame Alice Owen’s School and The Cardinal Vaughan Memorial RC School – both comprehensive – have high attainer populations of 79% and 77% respectively, while Fort Pitt Grammar School in Chatham, Kent and Skegness Grammar School are at 65% and 66% respectively.

This raises intriguing questions about the nature of selective education and the dividing line between selective and non-selective schools

At the other extreme there are no schools recording zero high attainers, but 99 record 10% or fewer, several of them prominent academies. This is an improvement on 2012 when 110 schools fell into this category.

The two schools with the fewest high attainers (3%) are Barnfield Business and Enterprise Studio (which opened in 2013) and St Aldhelm’s Academy in Poole.

Academies based in coastal towns are well represented amongst the 99.

It is interesting to speculate whether very small high attainer cohorts generally perform better than slightly larger cohorts that perhaps constitute a ‘critical mass’.

Certainly there are huge variations in performance on the key measures amongst those schools with few high attainers (where results have not been suppressed). This is particularly true of EBacc entry and success rates.

For example, amongst the 50 schools with the fewest high attainers:

  • the EBacc success rate varies from 73% at Aston Manor Academy to zero, returned by 12 of the 50.
  • The percentage of high attaining pupils making the expected progress in English varies from 50% to 100% while the corresponding range in maths is from 47% to 100%.

Many of these figures are derived from very small cohorts (all are between 1 and 20), but the point stands nevertheless.

.

The Performance of High Attainers

As noted above, there are no true high attainment measures relating to the achievement of GCSE A*/A grades within the Secondary Tables, so this section is necessarily reliant on the universal measures they contain.

.

5+ GCSEs at Grades A*-C including English and maths

The 2013 Secondary Performance Tables reveal that:

  • 53.6% of students at state-funded schools achieved 5+ GCSEs at Grades A*-C including English and maths, up 1.7% from 51.9% in 2012.
  • But the Tables pay more attention to the percentage achieving 5+ GCSEs at Grades A*-C (or equivalent) including GCSEs in English and maths: 60.6% of students attending state-funded schools achieved that measure in 2013, compared, up 1.8% from 58.8% in 2012.
  • 94.7% of high attainers in state-funded schools secured this outcome, up an improvement on 94.0% in 2012. The comparable figures for middle attainers and low attainers (with 2012 figures in brackets) are 57.4% (55.1%) and 6.9% (7.1%) respectively. Hence the overall increase of 1.8% masks a slight fall amongst low attainers and a significantly smaller increase amongst high attainers. Although there has been improvement, one in every 20 high attainers continues to fall short.
  • But it is notable that around 530 schools achieved 100% amongst their high attainers on this measure, compared with some 480 in 2012. Moreover, only 14 schools are at or below 67%, compared with 19 in 2012, and 47 are at or below 75% compared with 66 in 2012. This is positive news and suggests that the inclusion of the distinction within the Tables is beginning to bear fruit.

 Tables 2A and 2B, below show there has been an increase of 3.5% on this measure since 2011 across all pupils in state-funded mainstream schools. Meanwhile the proportion of high attainers securing this outcome has fallen by 0.4% (after rising slightly in 2012).

It may well be harder for schools to eradicate the last vestiges of underachievement at the top end than to strengthen performance amongst middle attainers, where there is significantly more scope for improvement. But some may also be concentrating disproportionately on those middle attainers.

This overall picture masks very different trends in different types of school.

In sponsored academies an overall improvement of 4.8% coincides with a slight 0.1% fall amongst high attainers, who have recovered following a substantial dip in 2012.

But in converter academies the overall success rate has fallen by almost 9% since 2011, while the rate for high attainers has fallen by only 2.7%.

And in free schools, UTCs and studios a slight overall fall since 2012 (there are no figures for 2011) is accompanied by an improvement for high attainers of over 4%.

Comprehensive schools have improved by 2.6% overall since 2011, yet their high attainers have fallen back by 0.3%. In selective schools the overall rate has fallen back by 2.3% while the high attainer rate has dropped by a similar 2.0%. This is concerning.

It is not straightforward to work out what is happening here, though the changing size of different sectors must be having a significant impact. 2012 GCSE results in English will certainly have influenced the dip in last year’s figures.

High attainers in free schools, UTCs and studios still have some ground to make up on other sectors and it will be interesting to see whether their improving trend will continue in 2014

 ,

Table 2A: Percentage achieving 5+ A*-C grades (or equivalent) including English and maths by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools
All HA All HA All HA All HA All HA
2013 61.7 94.7 59.2 94.1 51.2 93.0 68.2 95.7 54.6 91.7
2012 59.8 94.0 58.2 93.5 49.3 91.5 68.4 95.5 55.7 87.5
2011 58.2 95.1 N/A N/A 46.8 93.1 77.1 98.4 N/A N/A

.

Table 2B: Percentage achieving 5+ A*-C grades (or equivalent) including English and maths by admissions practice

Selective Comprehensive Modern
All HA All HA All HA
2013 96.4 97.3 60.4 94.5 55.3 92.5
2012 97.4 98.2 58.5 93.5 53.1 92.2
2011 98.7 99.3 57.8 94.8 50.8 91.8

.

A*-C grades in GCSE English and maths

According to the 2013 Secondary Tables:

  • 61.3% of all students in state-funded schools achieved GCSE grades A*-C in English and maths, compared with 59.5% in 2012, an improvement of 1.8%.
  • However, 95.1% of high attainers in state-funded schools achieved this measure compared with 94.3% in 2012, an increase of only 0.8%. The comparable figures for middle and low attainers (with 2012 figures in brackets) were 58.5% (55.8%) and 7.1% (7.3%) respectively. The pattern is therefore similar to the 5A*-C measure, with limited improvement at the top, significant improvement in the middle and a slight decline at the bottom.
  • Some 610 state-funded schools had 100% of their high attainers achieve this outcome, a significant improvement on the 530 recorded in 2012. There were 12 schools where the percentage was 67% or lower, compared with 18 in 2012, and 38 where the percentage was 75% or lower, compared with almost 60 in 2012.
  • These latter figures include Pate’s, King Edward VI Camp Hill and Bishop Wordsworth’s, all presumably tripped up again by their choice of IGCSE. Other poor performers were Gloucester Academy (again) at 44% and St Antony’s Catholic College in Trafford (59%). The worst performers were relatively stronger than their predecessors from 2012.

The trend data derived from the associated statistical publications shows that the overall figure for high attainers in state-funded schools has increased by 0.9% compared with 2012, recovering most of the 1.2% dip that year compared with 2011.

Sponsored academies have improved significantly with their high attainers back to 93.5% (their 2011 percentage) following a 1.7% dip in 2012. On the other hand, high attainers in converter academies have made little improvement compared with 2012, while free schools, studio schools and UTCs have improved by 3.9%.

Once again these patterns are probably influenced strongly by change in the size of some sectors and the impact of 2012 GCSE English results.

Interestingly though, selective school high attainers – having managed 99.5% on this measure in 2011 – are continuing to fall back, recording 98.3% in 2012 and now 97.5%. This may have something to do with the increasing attraction of IGCSE.

 .

Entry to and achievement of the EBacc

The 2013 Secondary Tables show that:

  • 35.5% of all students at state-funded schools were entered for all English Baccalaureate subjects, compared with 23.1% in 2012, and 22.8% achieved all EBacc subjects, up 6.6% from 16.2% in 2012.
  • Both entry (65.0%) and success (52.1%) rates continue to be much higher for high attainers than for middle attainers (27.8% entered and 11.8% successful) and low attainers (3.4% entered and 0.5% successful)
  • In 2012, the entry rate for high attainers was 46.3%, so there has been a substantial improvement of almost 19%. The 2012 success rate was 38.5%, so that has improved by 13.6%.
  • One could reasonably argue that a 52.1% success rate is lower than might be expected and relatively disappointing given that almost two-thirds of high attainers now enter all EBacc subjects. But, compared with the two previous measures, schools are much further away from the 100% ceiling with the EBacc, so further significant improvement amongst high attainers is likely over the next few years. However, the forthcoming shift to ‘Progress 8’ measure is likely to impact significantly.
  • 55 schools entered no high attainers for the EBacc, down considerably from 186 in 2012. Zero high attainers achieved the EBacc at 79 schools compared with 235 in 2012. Several grammar schools were of this number.

Tables 3A and B below indicate that, despite rapid improvement since 2012, only a third of high attainers in sponsored academies achieve the EBacc, compared with almost 6 in 10 attending converter academies.

The success rate for high attainers at free schools, UTCs and studios is only slightly higher than that for sponsored academies and both are improving at a similar rate.

Amost exactly half of high attainers at comprehensive schools are successful, as are almost exactly three quarters of high attainers at selective schools, but the rate of improvement is much faster in comprehensive schools – and indeed in modern schools too.

.

Table 3A: Percentages achieving the EBacc by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools
All HA All HA All HA All HA All HA
2013 23.2 52.1 20.9 49.1 11.0 34.7 30.1 58.1 16.1 35.6
2012 16.4 38.5 14.5 35.0 6.3 21.1 25.7 49.1 12.2 23.6
2011 15.6 37.2 - - 5.2 17.7 31.5 55.4 - -

.

Table 3B: Percentages achieving the EBacc by admissions practice

Selective Comprehensive Modern
All HA All HA All HA
2013 71.6 74.6 21.5 49.9 12.1 33.3
2012 68.2 70.7 14.5 35.0 7.2 20.7
2011 68.1 70.5 13.7 33.6 6.7 20.3

.

Three Levels of Progress in English and maths

The Tables inform us that:

  • 70.4% of all pupils in state-funded secondary schools made at least three levels of progress in English (up 2.4% from 68% in 2012) and 70.7% did so in maths (up 2.0% from 68.7% in 2012).
  • In both subjects more high attainers made the requisite progress than middle and low attainers: 86.2% in English (2.8% up on 2012) and 87.8% in maths (up 2.0%). Despite these improvements, it remains the case that approximately one in seven high attainers fail to make the expected progress in English and one in eight fail to do so in maths. This is extremely disappointing.
  • There were 108 schools in which every high attainer made the requisite progress in English, up from 93 in 2012. In maths, 120 schools ensured every high attainer made the expected progress, compared with 100 in 2012. A total of 36 schools managed this feat in both English and maths, whereas only 26 did so in 2012.
  • At the three grammar schools we have already encountered, no high attainers made the expected progress in English. Forty-four schools were at or below 50% on this measure, down markedly from 75 in 2012. The worst performer apart from the grammar schools was Gloucester Academy at 28%.
  • Thirty-one schools had 50% or fewer high attainers making the expected progress in maths, an improvement on the 46 registering this result last year. The poorest performer was Stafford Sports College at 29%.

Tables 4A and B below contain the trend data for the achievement of three levels of progress in English while Tables 5A and B cover maths.

The figures within these tables are not strictly comparable, since the statistics unaccountably define high attainment slightly differently for the two populations. In the case of the ‘all’ column, they use achievement of Level 5 in the relevant KS2 test (ie English or maths), rather than above Level 4 achievement across all three core subjects, while the definition for the ‘high attainers’ column is the customary one set out above.

Nevertheless, one can see that, overall, the percentage of high attainers meeting this benchmark in English is recovering following a significant fall last year. Free schools, UTCs and studios have just overtaken sponsored academies while converter academies are 3.5 percentage points ahead of LA maintained mainstream schools.

In maths converter academies have an even more substantial 4.5 percentage point lead over LA maintained mainstream schools. Sponsored academies are a full 10 percentage points behind converters and five percentage points behind free schools, UTCs and studios, but the latter category is recording a downward trend while everyone else is moving in the opposite direction.

The fact that one in five high attainers in sponsored academies, free schools, UTCs and studios is failing to make three levels of progress in English is serious cause for concern.  Worse still, the same is true of maths in sponsored academies. This state of affairs requires urgent attention.

It is noticeable that the general recovery in performance amongst high attainers in English does not extend to selective schools, which have fallen back still further since 2012 and are now a full 3.5 percentage points behind their 2011 level. Regardless of causality – and early entry policy as well as the increasing popularity of IGCSE may be involved – this too is a matter for concern. The situation is more positive in maths however.

.

Table 4A: Percentages achieving three levels of progress in English by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools
All HA All HA All HA All HA All HA
2013 79.7 86.2 - 85.0 - 80.7 - 88.5 - 81.0
2012 76.9 83.4 - 82.5 - 76.0 - 86.7 - 68.1
2011 69.0 87.2 - - - 79.6 - 94.5 - -

.

Table 4B: Percentages achieving three levels of progress in English by admissions practice

Selective Comprehensive Modern
All HA All HA All HA
2013 - 93.0 - 85.5 - 81.0
2012 - 93.4 - 82.3 - 77.1
2011 - 96.5 - 86.2 - 81.1

.

Table 5A: Percentages achieving three levels of progress in maths by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools
All HA All HA All HA All HA All HA
2013 81.7 87.8 - 86.2 - 80.8 - 90.7 - 85.6
2012 79.7 85.8 - 84.4 - 77.8 - 90.2 - 87.5
2011 76.8 85.2 - - - 75.6 - 93.2 - -

.

Table 5B: Percentages achieving three levels of progress in maths by admissions practice

Selective Comprehensive Modern
All HA All HA All HA
2013 - 96.6 - 86.9 - 84.4
2012 - 95.9 - 84.7 - 80.5
2011 - 96.6 - 83.9 - 79.3

.

Other measures

The Performance Tables show:

  • The average point score (APS) per pupil for the best eight subjects (GCSE only) across all state-funded schools was 280.1, up from 276.7 in 2012. Amongst high attainers this rose to 377.6, up from 375.4 in 2012. Only five schools – all selective – achieved an APS above 450 for their high attainers (eight schools managed this in 2012). The top performer was Colyton Grammar School in Devon. At the other extreme, four schools were at 200 or lower (much reduced from 16 in 2012). These were Hadden Park High School in Nottingham (160.1), Pent Valley Technology College in Kent (188.5), Aylesford School in Kent (195.7) and Bolton St Catherine’s Academy (198.2).
  • According to the value added (best 8) measure, the best results for high attainers were achieved by four schools that scored over 1050 (seven schools managed this in 2012). These were Tauheedul Islam Girls High School, Harris Girls’ Academy, East Dulwich, Sheffield Park Academy and Lordswood Girls’ School and Sixth Form Centre. Conversely there were three schools where the score was 900 or below. The lowest VA scores were recorded by Ark Kings Academy; Hadden Park High School; and Manchester Creative and Media Academy for Boys.
  • The Tables also provide an average grade per GCSE per high attainer (uncapped) but, at the time of writing, the relevant column in the Tables refuses to sort in ascending/descending order. This press article draws on another measure – average grade per pupil per qualification (capped at best 8) to identify Colyton Grammar School as the only state-funded school to achieve an average A* on this measure. It is highly likely that Colyton Grammar will top the rankings for the uncapped high attainer measure too. The article adds that a total of 195 schools (state and independent presumably) achieved an average of either A*, A*-, A+, A or A- on the capped measure, noting that the Hull Studio School posted G- (though with only seven pupils) while two further schools were at E+ and a further 82 schools averaged D grades.
  •  The average number of GCSE entries for high attainers in state-funded schools was 9.9, up slightly from 9.7 in 2012. The highest level of GCSE entries per high attainer was 15.5 at Colyton Grammar School, repeating its 2012 achievement in this respect. At three schools – Hadden Park, Aylesford and Bolton St Catherine’s – high attainers were entered for fewer than five GCSEs (15 schools were in this category last year). One school – Ormesby in Middlesbrough – entered its high attainers for 22 qualifications, which seems a little excessive.

 

A Level Achievement

.

Percentage achieving 3+ A levels at AAB+ in facilitating subjects

According to the 16-18 Performance Tables:

  • 7.5% of A level students in all state funded schools and colleges achieved three A levels, all in facilitating subjects, at AAB or higher, up slightly from 7.4% in 2012. This is another column in the Tables that – at the time of writing – will not sort results into ascending/descending order. In 2012 a handful of state-funded institutions achieved 60% on this measure and that is likely to have been repeated. There were also 574 schools and colleges that recorded zero in 2012 and there may have been a slight improvement on that this year.
  • The same problem arises with the parallel measure showing the percentage of students achieve AAB+ with at least two in facilitating subjects. We know that 12.1% of A level students in state funded schools and colleges achieved this, up very significantly from 7.8% in 2012, but there is no information about the performance of individual schools. In 2012 a handful of institutions achieved over 80% on this measure, with Queen Elizabeth’s Barnet topping the state schools at 88%. At the other extreme, there were about 440 schools and colleges which recorded zero in 2012.

Tables 6A and B below show that the success rate on the first of these measures is creeping up in LA-maintained mainstream schools, sponsored academies, sixth form colleges and FE colleges. The same is true of comprehensive schools.

On the other hand, the success rate is falling somewhat in converter academies, free schools, UTCs and studios – and also in selective and modern schools.

It is noticeable how badly sponsored academies fare on this measure, achieving exactly half the rate of LA-maintained mainstream schools.

.

Table 6A: Percentages of students achieving 3+ A levels at AAB+ in facilitating subjects by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools Sixth form colleges FE
2013 8.7 7.4 3.7 10.4 5.1 6.0 3.5
2012 8.6 7.2 3.4 11.4 7.5 5.8 3.3
2011 - - - - - - -

.

Table 6B: Percentages of students achieving 3+ A levels at AAB+ in facilitating subjects in schools by admissions practice

Selective Comprehensive Modern
2013 21.1 6.8 1.0
2012 21.5 6.6 1.6
2011 - - -

.

The 2013 statistics also contain breakdowns for the ‘AAB+ with two in facilitating subjects’ measure, as shown in Table 6C below.

.

Table 6C: Percentages of students achieving 3+ A levels at AAB+ with two in facilitating subjects by sector and admissions practice, 2013 only.

State-funded mainstream schools 13.6
LA-funded mainstream schools 11.4
Sponsored academies (mainstream) 5.4
Converter academies (mainstream) 16.4
Mainstream free schools, UTCs and studios 11.3
Sixth Form Colleges 10.4
FE Colleges 5.8
Selective schools 32.4
Comprehensive schools 10.7
Modern schools 2.0

.

While sponsored academies are achieving unspectacular results – in that they are even further behind LA-funded schools and even below FE colleges on this measure – selective schools are managing to get almost one third of their students to this level.

.

Percentage achieving 3+ A levels at A*/A

The Performance Tables do not include this measure, but it is included in the statistical reports. Tables 7A and B below show the trends – downwards in all sectors and types of school except FE colleges and free schools, UTCs and studios.

It is unclear why their performance should be improving on this measure but declining on the AAB+ in three facilitating subjects measure, though it seems likely that some UTCs and studios are less likely to enter their students for facilitating subjects.

.

Table 7A: Percentages of all students achieving 3+ A levels at A*/A by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools Sixth form colleges FE
2013 10.7 8.7 4.1 13.1 7.9 9.3 5.1
2012 10.9 9.1 4.2 14.8 6.0 9.7 5.0
2011 11.4 - - - - 10.2 4.9

.

Table 7B: Percentages of students achieving 3+ A levels at A*/A in schools by admissions practice

Selective Comprehensive Modern
2013 27.0 8.1 1.7
2012 27.7 8.3 1.9
2011 27.7 8.4 2.3

.

Percentage achieving 3+ A levels at AAB+

Once again, this measure is not in the Tables but is in the statistical bulletin. Tables 8A and B below compare trends. The broad trend is again downwards although this is being bucked (just) by FE colleges and (more significantly) by sponsored academies. So while sponsored academies are getting slightly falling results at AAA+, their results are improving at AAB.

 .

Table 8A: Percentages of students achieving 3+ A levels at AAB+ by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools Sixth form colleges FE
2013 17.9 15.1 7.9 21.4 13.0 16.4 9.5
2012 17.9 15.4 7.5 23.4 16.4 16.8 9.4
2011 - - - - - - -

.

Table 8B: Percentages of students achieving 3+ A levels at AAB+ in schools by admissions practice

Selective Comprehensive Modern
2013 40.0 14.5 4.4
2012 40.6 14.5 4.7
2011 40.9 14.8 5.4

It is interesting to compare 2013 performance across these different high attainment measures and Table 9 below does this, enabling one to see more clearly the differentiated response to facilitating subjects amongst high attainers.

In most parts of the schools sector, the success rate for AAB+ in any subjects is roughly twice that of AAB+ in facilitating subjects, but this is not true of FE and sixth form colleges, nor of free schools, UTCs and studios.

At the same time, every sector and school type shows a higher rate at AAA+ than at AAB+ in three facilitating subjects.

.

Table 9: Percentages of students by sector/school type achieving different A level high attainment measures in 2013

AAB+ in 3FS AAB+ 2in FS AAB+ AAA+
All state-funded mainstream 8.7 13.6 17.9 10.7
LA-funded mainstream 7.4 11.4 15.1 8.7
Sponsored academies 3.7 5.4 7.9 4.1
Converter academies 10.4 16.3 21.4 13.1
Free schools UTCs and studios 5.1 11.3 13.0 7.9
Sixth form colleges 6.0 10.4 16.4 9.3
FE colleges 3.5 5.8 9.5 5.1
Selective schools 21.1 32.4 40.0 27.0
Comprehensive schools 6.8 10.7 14.5 8.1
Modern schools 1.0 2.0 4.4 1.7

.

Other Measures

Reverting back to the Performance Tables:

  • The APS per A level student across all state-funded institutions is 782.3, up significantly from 736.2 in 2012. The highest APS was recorded by Dartford Grammar School which recorded 1650.0. At the other end of the spectrum, an APS of 252.6 was recorded by Hartsdown Technology College in Kent.
  • The APS per A level entry across all state-funded institutions was 211.3 compared with 210.2 in 2012. The strongest performer in the maintained sector was Queen Elizabeth’s School, Barnet, which achieved 271.4. The lowest score in the maintained sector is 97.7, at Appleton Academy in Bradford.
  • A new average point score per A level pupil expressed as a grade is dominated by independent schools, but the top state-funded performers – both achieving an average A grade – are Henrietta Barnet and Queen Elizabeth’s Barnet. A handful of schools record U on this measure: Appleton Academy, The Gateway Academy in Thurrock, Hartsdown Technology College and The Mirus Academy in Walsall.
  • A new A level value added measure has also been introduced for the first time. It shows Ripon Grammar School as the top performer scoring 0.61. The lowest score generated on this measure is -1.03 at Appleton Academy, which comes in comfortably below any of its competitors.
  • The Statistical Bulletin also tells us what percentage of A level entries were awarded A* and A grades. Tables 10 A and B below record this data and show the trend since 2011. It is evident that A* performance is falling back slightly in every context, with the sole exception of FE (a very slight improvement) and free schools, UTCs and studios. A*/A performance is generally holding up better, other than in converter academies.

 .

Table 10A: Percentage of A* and A*/A grades by sector

All state-funded mainstream LA maintained mainstream Sponsored academies Converter academies Free schools, UTCs and studio schools Sixth form colleges FE
A* A*/A A* A*/A A* A*/A A* A*/A A* A*/A A* A*/A A* A*/A
2013 6.8 24.4 5.9 21.4 3.7 15.1 7.9 27.5 4.7 20.0 5.7 21.6 3.9 15.4
2012 7.2 24.3 6.3 21.8 4.0 14.4 8.9 29.1 4.3 20.3 5.8 21.8 3.8 15.6
2011 7.3 24.5 - - - - - - - - 6.3 22.4 3.9 16.0

.

Table 10B: Percentage of A* and A*/A grades in schools by admissions practice

Selective Comprehensive Modern
A* A*/A A* A*/A A* A*/A
2013 12.7 40.6 5.6 21.1 2.5 10.6
2012 13.4 41.2 5.9 20.9 2.6 11.1
2011 13.4 41.0 6.0 21.1 3.0 11.6

.

Conclusion

What are we to make of this analysis overall?

The good news is that high attainers have registered improvements since 2012 across all the most important GCSE measures:

  • 5+ GCSEs or equivalent including English and maths GCSEs (up 0.7%)
  • GCSEs in English and maths (up 0.8%)
  • 3 levels of progress in English (up 2.8%)
  • 3 levels of progress in maths (up 2.0%)
  •  EBacc entry (up 19%) and EBacc achievement (up 13.6%).

The last of these is particularly impressive.

At A level, the underlying trend for high attainment per se is slightly downward, but significantly upward for AAB+ grades with two in facilitating subjects.

The less good news is that some of these improvements have been made from a relatively low base, so the overall levels of performance still fall short of what is acceptable.

It is not cause for congratulation that one in seven high attainers still fail to make the expected progress in English, while one in eight still fail to do so in English. Nor is it encouraging that one in twenty high attainers still fail to secure five or more GCSEs (or equivalent) including GCSEs in English and maths.

The significant improvement with the EBacc masks the fact that one fifth of high attainers who enter exams in all the requisite subjects still fail to secure the necessary grades.

Moreover, some schools are demonstrating very limited capacity to secure high achievement and – in particular – sufficient progress from their high attainers.

The fact that several schools achieve better progression in English for both their middle and their low attainers is particularly scandalous and needs urgent attention.

As noted above, the floor targets regime is too blunt an instrument to address the shortcomings in the substantial majority of schools that have been highlighted in this study for poor performance on one or more high attainers measures.

The combined impact of the secondary accountability reforms planned for 2016 is as yet unclear. For the time being at least, Ofsted inspection is the only game in town.

The school inspection guidance now demands more attention to the performance and progress made by the ‘most able students’. But will inspection bring about with sufficient rapidity the requisite improvements amongst the poorest performers highlighted here?

I have it in mind to monitor progress in this small sample of twenty-two schools – and also to look back on what has happened to a parallel group of the poorest performers in 2012.

.

GP

January 2014

Gifted Phoenix’s 2013 Review and Retrospective

.

This final post of 2013 takes a reflective look back at this year’s activity.

4-Eyes-resized-greenjacketfinal

One purpose is straightforward self-congratulation – a self-administered pat on the back for all my hard work!

This is also an opportunity to review the bigger picture, to reflect on the achievements and disappointments of the year now ending and to consider the prospects for 2014 and beyond.

Perhaps I can also get one or two things off my chest…

…So, by way of an aside, let me mention here that I provide this information to you entirely free of charge, partly because I believe that global progress in (gifted) education is obstructed by the rationing of knowledge, partly to encourage those who construct and shelter behind paywalls to reflect on the negative consequences of their behaviour.

I try my best to offer you a factual, balanced and objective assessment, to flag up weaknesses as well as strengths. In short, I tell it like it is. I have no interest in self-aggrandisement, in reputation or the trappings of academia. You will search in vain for those trappings in my CV, but I speak and write with commensurate authority, based on extended experience as a national policy maker and student of the field …

Another purpose is to provide an annotated list of my posts, so that readers can catch up with anything they missed.

I make this my 35th post of 2013, five fewer than I managed in 2012. I took an extended break during August and September this year, half of it spent on tour in Western Australia and the remainder engaged on other projects.

During the course of the year I’ve made a conscious effort simultaneously to narrow and diversify my focus.

I’ve devoted around two-thirds of my posts to educational reform here in England, while the remainder continued to address global issues.

Some of the Anglocentric posts were intended to draw out the wider implications of these reforms, rather than confining themselves exclusively to gifted education and the impact on gifted learners.

I wanted to paint on a broader canvas. It is all too easy to exist in a gifted education ghetto, forgetting that it must be integral to our national educational systems as well as a global endeavour in its own right.

 .

Global Gifted Education

During 2013 I published two feature-length posts about the performance of high achievers in international comparisons studies:

Like it or not, these international tests are becoming increasingly influential in most countries around the world. Those involved in gifted education ignore them at their peril.

Many of the countries that top the rankings already invest significantly in gifted education – and some of those that do not (invest significantly and/or top the rankings) ought seriously to consider this as a potential route to further improvement.

Other posts with a global gifted focus include:

My best effort at a personal credo, derived from the experience of writing this Blog. Colleagues were very flattering

.

.

I supplemented the post with a vision for delivery, primarily to inform UK-based discussion within GT Voice, but also relevant to Europe (the EU Talent Centre) and globally (the World Council).

I took a second look at this nascent field, exploring developments since I first blogged about it in 2010. I like to flatter myself that I invented the term.

The post tells of the passing interest exhibited by IRATDE and notes the reference in the July 2012 World Council Newsletter to a special issue of Gifted and Talented International (GTI) that will be devoted to the topic.

I heard in May that an unnamed specialist had been invited to prepare a ‘target paper’, but nothing has materialised to date. The wheels of academic publishing turn parlous slow.

I concluded the post with a tongue-in cheek contribution of my own – the Gifted Phoenix Equation!

Minimising the Excellence Gap and Optimising the Smart Fraction maximises impact on Economic Growth (Min EG + Optimal SF = Max EG)

This post opened with a self-confessed rant about the ‘closed shop’ operated by academics in the field, defended by research paywalls and conference keynote monopolies.

But I set aside my prejudices to review the nine leading academic journals in gifted education, examine the rights the publishers offer their authors and offer a constructive set of proposals for improving the accessibility of research.

There were also a handful of new national studies:

the last of which is strictly a transatlantic study of support for low income high ability students, developed from analysis of the US NAGC publication of the same name.

.

Gifted Education in England

Two posts examined material within England’s national school performance tables relating to high attainment and high attainers.

The latter is the second such analysis I have provided, following one on the 2012 Tables published last December. The former will be supplanted by a new version when the Secondary Tables are published in January.

I also offered a detailed treatment of the underlying accountability issues in:

These posts explored the rather haphazard treatment now afforded ‘the most able students’ in documents supporting the School Inspection Framework, as well as the different definitions deployed in the Performance Tables and how these might change as a consequence of the trio of accountability consultations launched this year.

.

.

During the Spring I wrote:

Despite the Government’s reported intention to establish a national network of up to twelve of these, still only two have been announced – sponsored by King’s College London and Exeter University respectively.

I might devote a 2014 post to updating my progress report.

There was also special mini-series, corralled under the speculatively optimistic title: A Summer of Love for Gifted Education?’

This is fundamentally a trilogy:

The original conceit had been to build each episode around a key publication expected during the year. Episodes One and Two fitted this description but the third, an ‘Investigation of school- and college- level strategies to raise the Aspirations of High-Achieving Disadvantaged Pupils to pursue higher education’ was (is still) overdue, so I had to adjust the focus.

Episode Two was a particularly rigorous examination of the Ofsted report that led to the changes to the inspection documentation.

.

.

In Episode Three, I took the opportunity to expose some questionable use of statistics on the part of selective universities and their representative bodies, setting out a 10-point plan to strengthen the representation of disadvantaged students at Oxford and Cambridge. This was accompanied by a flying pig.

.

.

There were also some supplementary posts associated with the Summer of Love:

And some material I produced at the time that Ofsted published ‘The Most Able Students’:

Did it turn out to be a ‘Summer of Love’? Looking back now, I have mixed feelings. Significant attention was paid to meeting the needs of high attaining learners, and those needs are likely to be better recognised and responded to as a consequence.

But the response, such as it is, relies almost exclusively on the accountability system. There is still a desperate need for authoritative updated national framework guidance. Ideally this should be developed by the national gifted education community, working collaboratively with government seed funding.

But the community shows little sign of readiness to take on that responsibility. Collaboration is virtually non-existent:  GT Voice has failed thus far to make any impact (justifying my decision to stand down from the board in protest at frustratingly slow progress).

Meanwhile, several players are pursuing their own diverse agendas. Most are prioritising income generation, either to survive or simply for commercial gain. Everyone is protecting their corner. Too many scores are being settled. Quality suffers.

For completeness, I should also mention a couple of shorter posts:

a piece I wrote for another publisher about how free schools might be rolled into this national collaborative effort, and

which was my best effort to summarise the ‘current state’ on the other side of Ofsted’s Report, as well as an alternative future vision, avoiding the Scylla of top-down centralised prescription and the Charybdis of bottom-up diffused autonomy.

 

Wider English Educational Reform

Almost all the posts I have written within this category are associated with emerging national policy on curriculum and assessment:

.

.

There was even

which I still expect to see in a manifesto come 2015!

As things stand, there are still many unanswered questions, not least where Labour stands on these issues.

Only one of three accountability consultations has so far received a Government response. The response to the primary consultation – comfortably the least persuasive of the three – was due in ‘the autumn’ but hadn’t appeared by Christmas.

The decision to remove National Curriculum levels looks set to have several unintended negative consequences, not least HMCI Wilshaw’s recent call for the reintroduction of national testing at KS1 and KS3.

I am still to be persuaded that this decision is in the best interest of high attainers.

 

Social Media

This year I have spent more time tweeting and less time producing round-ups of my Twitter activity.

At the time of writing, my follower count has reached 4,660 and I have published something approaching 18,700 Tweets on educational topics.

I try to inform my readers about wider developments in UK (especially English) education policy, keeping a particularly close eye on material published by the Government and by Parliament.

I continue to use #gtchat (global) and #gtvoice (UK) to hashtag material on gifted education and related issues. I look out particularly for news about developments worldwide. I publish material that seems interesting or relevant, even though I might disagree with it. I try to avoid promotional material or anything that is trying to sell you something.

I began 2013 intending to produce round-ups on ‘a quarterly-cum-termly basis’ but have managed only two editions:

The next volume is already overdue but I simply can’t face the grinding effort involved in the compilation process. I may not continue with this sequence in 2014.

I was also invited to answer the question:

ResearchED was a conference organised via Twitter which took place in September.

The post argued for a national network of UK education bloggers. This hasn’t materialised, although the status and profile of edublogging has improved dramatically during 2013, partly as a consequence of the interest taken by Michael Gove.

There are many more blogs and posts than a year ago, several co-ordinated through Blogsync and/or reblogged via The Echo Chamber.

Precious few bloggers enter the field of gifted education, though honourable mentions must go to Distilling G&T Ideas and Headguruteacher.

Elsewhere in the world, not too many gifted education bloggers are still generating a constant flow of material.

Exceptions include Lisa Conrad, who is maintaining two blogs in the US Gifted Parenting Support and Global #GT Chat Powered by TAGT. Also Kari Kolberg who produces Krummelurebloggen (in Norwegian) and Javier Touron who writes Talento y Educacion (in Spanish).

I need urgently to revisit my Blogroll. I might also write a post about the general state of global gifted education blogging in the early part of 2014.

 

Reference

I have made only limited progress this year with the reference pages on this Blog:

  • Who’s Who?  remains embryonic. I had plans to force myself to produce a handful of entries each day, but managed only two days in succession! There isn’t a great deal of intellectual challenge in this process – life may be too short!
  • Key Documents is a mixed bag. The UK pages are fully stocked. You should be able to find every significant national publication since 2000. The Rest of the World section is still largely empty.

Rightly or wrongly, the production of blog posts is taking priority.

 

Analytics

Compared with 2012, the number of page views has increased by over 30%, although the number of posts is down by 12.5%. I’m happy with that.

Some 40% of views originate in the UK. Other countries displaying significant interest include the US, Singapore, Australia, India, Hong Kong, Saudi Arabia, New Zealand, Canada and Spain. Altogether there have been visits from 169 countries.

The most popular posts published this year are, in order of popularity:

  • Whither National Curriculum Assessment Without Levels?
  • What the KS2/KS4 Transition Matrices Show About High Attainers’ Performance
  • High Attaining Students in the 2012 Secondary School Performance Tables
  • Analysis of the Primary Assessment and Accountability Consultation Document and
  • A Summer of Love for English Gifted Education Episode 2: Ofsted’s ‘The Most Able Students’

.

Visuals

I have changed the theme of my Blog twice this year – initially to Zoren and more recently to Highwind. I wanted a clearer, spacier look and a bigger font.

During the course of the year I have alternated between using my photographs within posts and producing work that is largely free of illustration. I have mixed feelings about this.

It seems somehow incongruous to intersperse unrelated photographs within a post about educational matters, but the stock of education-relevant non-copyrighted illustration is severely limited. Then again, screeds of unbroken text can be rather dreary to the eye.

So readers can expect some more views of Western Australia (especially) during 2014! Here’s one to whet your appetite.

.

Flora 2 by Gifted Phoenix

Flora 2 by Gifted Phoenix

 

The Future

I close 2013 in a pessimistic mood. Despite the more favourable domestic policy climate, I am markedly less optimistic about the future of gifted education than I was at the start of the year.

Disillusion is setting in, reinforced by negligible progress towards the objectives I hold most dear.

The ‘closed shop’ is as determinedly closed as ever; vested interests are shored up; governance is weak. There is fragmentation and vacuum where there should be inclusive collaboration for the benefit of learners. Too many are on the outside, looking in. Too many on the inside are superannuated and devoid of fresh ideas.

Every so often I witness dispiriting egotism, duplicity or even vengefulness. Disagreements fester because one or both of the parties is unwilling to work towards resolution.

The world of gifted education is often not a happy place – and while it remains that way there is no real prospect of achieving significant improvements in the education and life chances of gifted learners.

To mix some metaphors, it may soon be time to cut my losses, stop flogging this moribund horse and do something else instead.

Happy New Year!

.

GP

December 2013

PISA 2012: International Comparison of High Achievers’ Performance

.

This post examines what PISA 2012 can tell us about the comparative performance of high achievers in England, other English-speaking countries and those that top the PISA rankings.

Introductory Brochure for PISA 2012 by Kristjan Paur

Introductory Brochure for PISA 2012 by Kristjan Paur

It draws on a similar range of evidence to that deployed in my post on the PISA 2009 results (December 2010).

A more recent piece, ‘The Performance of Gifted High Achievers in TIMSS, PIRLS and PISA’ (January 2013) is also relevant.

The post reviews:

  • How the PISA 2012 Assessment Framework defines reading, mathematical and scientific literacy and its definitions of high achievement in each of the three core domains.
  • How average (headline) performance on the three core measures has changed in each jurisdiction compared with PISA 2006 and PISA 2009.
  • By comparison, how high achievers’ performance – and the balance between high and low achievers’ performance – has changed in each jurisdiction over the same period.
  • How jurisdictions compare on the ‘all-rounder’ measure, derived from achievement of a high performance threshold on all three assessments.

The twelve jurisdictions included in the main analysis are: Australia, Canada, England, Finland, Hong Kong (China), Ireland, New Zealand, Shanghai (China), Singapore, South Korea, Taiwan and the USA.

The post also compares the performance of the five home countries against the high achievement thresholds. I have foregrounded this analysis, which appears immediately below, save only for the headline (but potentially misleading) ‘top 10’ high achiever rankings for 2012.

.

Headlines

 .

World Leaders against PISA’s High Achievement Benchmarks

The top 10 performers in PISA 2012 against the high achievement benchmarks (Level 5 and above), in reading, maths and science respectively, are set out in Table 1 below.

The 2009 rankings are shown in brackets and the 2012 overall average rankings in bold, square brackets. I have also included England’s rankings.

.

Table 1

Rank Reading Maths Science
1 Shanghai (1) [1] Shanghai (1) [1] Shanghai (1) [1]
2 Singapore (3) [3] Singapore (2) [2] Singapore (2) [3]
3 Japan (5) [4] Taiwan (4) [4] Japan (5) [4]
4 Hong Kong (9) [2] Hong Kong (3) [3] Finland (3) [5]
5 S. Korea (6) [5] S Korea (5) [5] Hong Kong (6) [2]
6 N Zealand (2) [13] Liechtenstein (13) [8] Australia (7) [16]
7 Finland (4) [6] Macao (15) [6] N Zealand (4) [18]
8 Canada (7=) [8] Japan (8) [7] Estonia (17) [6]
9 France (13) [21] Switzerland (6) [9] Germany (8) [12]
10 Belgium (10) [16] Belgium (9) [15] [15] Netherlands (9) [14]
England 19th (19) [23] England 24th (32) [25] England 11th  (12) [18]

 .

On the basis of these crude rankings alone, it is evident that Shanghai has maintained its ascendancy across all three domains.

Singapore has reinforced its runner-up position by overtaking New Zealand in reading. Hong Kong and Japan also make it into the top ten in all three domains.

Notable improvements in the rankings have been made by:

  • Japan, Hong Kong and France in reading
  • Liechtenstein and Macao in maths
  • Japan and Estonia in science

.

.

Jurisdictions falling down the rankings include:

  • Australia, New Zealand and Finland in reading
  • Finland and Switzerland in maths
  • Canada and New Zealand in science.

Those whose high achiever rankings significantly exceed their average rankings include:

  • New Zealand, France and Belgium in reading
  • Belgium in maths
  • Australia, New Zealand, Germany and the Netherlands in science

The only one of the top ten jurisdictions exhibiting the reverse pattern with any degree of significance is Hong Kong, in science.

On this evidence, England has maintained its relatively strong showing in science and a mid-table position in reading, but it has slipped several places in maths.

Comparing England’s rankings for high achievers with its rankings for average performance:

  • Reading 19th versus 23rd
  • Maths 24th versus 25th
  • Science 11th versus 18th

This suggests that England is substantively stronger at the top end of the achievement spectrum in science, slightly stronger in reading and almost identical in maths. (The analysis below explores whether this is borne out by the proportions of learners achieving the relevant PISA thresholds.)

Overall, these rankings suggest that England is a respectable performer at the top end, but nothing to write home about. It is not deteriorating, relatively speaking – with the possible exception of mathematics – but it is not improving significantly either. The imbalance is not atypical and it requires attention, but only as part of a determined effort to build performance at both ends.

.

Comparing the Home Countries’ Performance

Table 2 below shows how each home country has performed at Level 5 and above in each of the three core PISA assessments since 2006.

.

Table 2

  2012 Level 5+ 2009 Level 5+ 2006 Level 5+
  Read Maths Sci Read Maths Sci Read Maths Sci
England 9.1 12.4 11.7 8.1 9.9 11.6 9.2 11.2 14.0
N Ireland 8.3 10.3 10.3 9.3 10.3 11.8 10.4 12.2 13.9
Scotland 7.8 10.9 8.8 9.2 12.3 11.0 8.5 12.1 12.5
Wales 4.7 5.3 5.7 5.0 5.0 7.8 6.4 7.2 10.9
UK 8.8 11.9 11.1 8.0 9.9 11.4 9.0 11.2 13.8
OECD average 8.4 12.6 8.4 7.6 12.7 8.5 8.6 13.3 9.0

.

In 2012, England is ahead of the other home countries in all three domains. Northern Ireland is runner-up in reading and science, Scotland in maths. Wales is a long way behind the other four in all three assessments.

Only England tops the OECD average in reading. All the home countries fall below the OECD average in maths, though all but Wales are above it in science.

Compared with 2006, England’s performance has changed little in reading, increased somewhat in maths (having fallen back betweentimes) and fallen quite significantly in science.

In comparison, Northern Ireland is on a downward trend in all three domains, as is Scotland (though it produced small improvements in maths and reading in 2009). Wales has fallen back significantly in science, though somewhat less so in reading and maths.

It seems that none of the home countries is particularly outstanding when it comes to the performance of their high achievers, but England is the strongest of the four, while Wales is clearly the weakest.

A slightly different perspective can be gained by comparing high and low performance in 2012.

Table 3 below shows that the proportion of low achievers is comfortably larger than the proportion of high achievers. This is true of all the home countries and all subjects, though the difference is less pronounced in science across the board and also in Scotland. Conversely, the imbalance is much more significant in Wales.

 .

Table 3

2012 Reading Maths Science
  L5+6 L1+below L5+6 L1+below L5+6 L1+below
England 9.1 16.7 12.4 21.7 11.7 14.9
N Ireland 8.3 16.7 10.3 24.1 10.3 16.8
Scotland 7.8 12.5 10.9 18.2 8.8 12.1
Wales 4.7 20.6 5.3 29.0 5.7 19.4
UK 8.8 16.7 11.9 21.8 11.1 15.0
OECD average 8.4 8.4 12.6 23.0 8.4 17.8

.

The ‘tail’ in reading is significantly higher than the OECD average in all four countries but – with the exception of Wales – somewhat lower in science.

In maths, the ‘tail’ is higher than the OECD average in Wales and Northern Ireland, but below average in England and Scotland.

The average figures suggest that, across the OECD as a whole, the top and bottom are broadly balanced in reading, there is a small imbalance in science towards the bottom end and a more significant imbalance in maths, again towards the bottom end.

By comparison, the home countries have a major issue at the bottom in reading, but are less significantly out of line in maths and science.

Overall, there is some evidence here of a longish tail of low achievement, but with considerable variation according to country and domain.

The bottom line is that all of the home countries have significant issues to address at both the top and the bottom of the achievement distribution. Any suggestion that they need to concentrate exclusively on low achievers is not supported by this evidence.

.

Francois Peron National Park by Gifted Phoenix 2013

Francois Peron National Park by Gifted Phoenix 2013

.

Background to PISA

 .

What is PISA?

The Programme for International Student Assessment (PISA) is a triennial OECD survey of the performance of 15 year-old students which typically covers maths, science and reading. Science was the main focus in 2006, reading in 2009 and maths in 2012.

PISA 2012 also included a computer-based assessment of problem-solving and a financial literacy assessment. However, some jurisdictions did not participate in the problem-solving exercise owing to ‘technical issues’ and financial literacy was undertaken by some countries only, as an optional extra.

Fifty-eight jurisdictions took part in PISA 2006 and 74 in PISA 2009 (65 undertook the assessment in 2009 and a further nine did so in 2010).

To date, a total of 65 jurisdictions have also taken part in PISA 2012.

According to the OECD’s own FAQ:

  • PISA tests reading, mathematical and scientific literacy ‘in terms of general competencies, that is, how well students can apply the knowledge and skills they have learned at school to real-life challenges. PISA does not test how well a student has mastered a school’s specific curriculum.’
  • Student performance in each field is comparable between assessments – one cannot reasonably argue therefore that a drop in performance is attributable to a more difficult assessment.
  • Each participating jurisdiction receives an overall score in each subject area – the average of all its students’ scores. The average score among OECD countries is set at 500 points (with a standard deviation of 100 points).
  • Participating jurisdictions are ranked in each subject area according to their mean scores, but:

‘is not possible to assign a single exact rank in each subject to each country…because PISA tests only a sample of students from each country and this result is then adjusted to reflect the whole population of 15-year-old students in that country. The scores thus reflect a small measure of statistical uncertainty and it is therefore only possible to report the range of positions (upper rank and lower rank) within which a country can be placed.’

Outside the confines of reports by the OECD and its national contractors, this is honoured more in the breach than the observance.

  • Scores are derived from scales applied to each subject area. Each scale is divided into levels, Level 1 being the lowest and Level 6 typically the highest

Further background detail on the 2012 assessments is set out in the ‘PISA 2012 Assessment and Analytical Framework’ (2013).

This explains that the framework for assessing maths was completely revised ahead of the 2012 cycle and ‘introduces three new mathematical processes that form the basis of developments in the reporting of PISA mathematics outcomes’, whereas those for science and reading were unchanged (the science framework was revised when it was the main focus in 2006 and ditto for reading in 2009).

The Framework clarifies the competency-based approach summarised in the FAQ:

‘ISA focuses on competencies that 15-year-old students will need in the future and seeks to assess what they can do with what they have learnt – reflecting the ability of students to continue learning throughout their lives by applying what they learn in school to non-school environments, evaluating their choices and making decisions. The assessment is informed, but not constrained, by the common denominator of national curricula. Thus, while it does assess students’ knowledge, PISA also examines their ability to reflect, and to apply their knowledge and experience to real-life issues in a reflective way. For example, in order to understand and evaluate scientific advice on food safety, an adult would need not only to know some basic facts about the composition of nutrients, but also to be able to apply that information.’

It explains that between 4,500 and 10,000 students drawn from 150 schools are typically tested in each jurisdiction.

Initial reports suggested that England would not take part in the 2012 assessments of problem-solving and financial literacy, but it subsequently emerged that this decision had been reversed in respect of problem-solving.

.

Setting PISA Outcomes in Context

There are plenty of reasons why one should not place excessive weight on PISA outcomes:

  • The headline rankings carry a significant health warning, which remains important, even though it is commonly ignored.

‘As the PISA 2000 and PISA 2003 samples for the United Kingdom did not meet the PISA response-rate standards, no trend comparisons are possible for these years.’ (p.1)

Hence, for the UK at least, reliable comparisons with pre-2006 results are off the table.

‘The pressure from policymakers for advice based on PISA interacts with this unhealthy mix of policy and technical people. The technical experts make sure that the appropriate caveats are noted, but the warnings are all too often ignored by the needs of the policy arm of PISA. As a result, PISA reports often list the known problems with the data, but then the policy advice flows as though those problems didn’t exist. Consequently, some have argued that PISA has become a vehicle for policy advocacy in which advice is built on flimsy data and flawed analysis.’

  • PISA is not the only game in town. TIMSS and PIRLS are equally significant, though relatively more focused on content knowledge, whereas PISA is primarily concerned with the application of skills in real life scenarios.
  • There are big political risks associated with worshipping at the PISA altar for, if the next set of outcomes is disappointing, the only possible escape route is to blame the previous administration, a strategy that wears increasingly thin with the electorate the longer the current administration has been in power.

 .

.

It would be quite wrong to dismiss PISA results out of hand, however. They are a significant indicator of the comparative performance of national (and regional) education systems. But they are solely an indicator, rather than a statement of fact.

.

What is assessed – and what constitutes high achievement – in each domain

The Assessment and Analytical Framework provides definitions of each domain and level descriptors for each level within the assessments.

.

Mathematical Literacy

The PISA 2012 mathematics framework defines mathematical literacy as:

‘An individual’s capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to recognise the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective citizens.’

Three aspects of maths are identified:

  • Mathematical processes and the fundamental capabilities underlying them. Three processes are itemised: formulating situations mathematically; employing mathematical concepts, facts, procedures and reasoning; and interpreting, applying and evaluating mathematical outcomes. The capabilities are: communication; mathematizing (transforming a real life problem to a mathematical form); representation; reasoning and argument; devising problem-solving strategies; using symbolic, formal and technical language and operations; and using mathematical tools.
  • Content knowledge, comprising four elements: change and relationships; space and shape; quantity; and uncertainty and data.
  • The contexts in which mathematical challenges are presented: personal; occupational; societal and scientific.

Six levels are identified within the PISA 2012 mathematics scale’. The top two are described thus:

  • ‘At Level 6 students can conceptualise, generalise and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply their insight and understandings along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments and the appropriateness of these to the original situations.’
  • ‘At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare and evaluate appropriate problem-solving strategies for dealing with complex problems related to these models. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal characterisations and insight pertaining to these situations. They can reflect on their actions and formulate and communicate their interpretations and reasoning.’

.

Reading literacy

Reading Literacy is defined as:

‘An individual’s capacity to understand, use, reflect on and engage with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.’

The assessment ‘is built on three major task characteristics’:

  • Situation – the context or purpose for which reading takes place, which may be personal (practical and intellectual interests), public (activities and concerns of society), educational (for learning purposes) or occupational (accomplishment of a task).
  • Text – the range of material that is read, which may be print or digital. In the case of digital text, the environment may be authored (the reader is receptive), message based, or mixed. In the case of both print and digital text, the format may be continuous (sentences and paragraphs), non-continuous (eg graphs, lists), mixed or multiple, while the text type may be description, narration, exposition, argumentation, instruction or transaction.
  • Aspect – how readers engage with the text, which includes accessing and retrieving; integrating and interpreting; and reflecting and evaluating.

Separate proficiency scales are provided for print and digital reading respectively. Both describe achievement in terms of the task rather than the student.

The print reading scale has six levels (Level One is subdivided into two). The top levels are described as follows:

  • Level 6: Tasks at this level typically require the reader to make multiple inferences, comparisons and contrasts that are both detailed and precise. They require demonstration of a full and detailed understanding of one or more texts and may involve integrating information from more than one text. Tasks may require the reader to deal with unfamiliar ideas, in the presence of prominent competing information, and to generate abstract categories for interpretations. Reflect and evaluate tasks may require the reader to hypothesise about or critically evaluate a complex text on an unfamiliar topic, taking into account multiple criteria or perspectives, and applying sophisticated understandings from beyond the text. A salient condition for access and retrieve tasks at this level is precision of analysis and fine attention to detail that is inconspicuous in the texts.
  • Level 5: Tasks at this level that involve retrieving information require the reader to locate and organise several pieces of deeply embedded information, inferring which information in the text is relevant. Reflective tasks require critical evaluation or hypothesis, drawing on specialised knowledge. Both interpretative and reflective tasks require a full and detailed understanding of a text whose content or form is unfamiliar. For all aspects of reading, tasks at this level typically involve dealing with concepts that are contrary to expectations.

For digital reading there are only four levels, categorised as 2-5. Level 5 is described thus:

‘Tasks at this level typically require the reader to locate, analyse and critically evaluate information, related to an unfamiliar context, in the presence of ambiguity. They require generating criteria to evaluate the text. Tasks may require navigation across multiple sites without explicit direction, and detailed interrogation of texts in a variety of formats.’

 .

Scientific literacy

Scientific literacy is defined as:

‘An individual’s scientific knowledge and use of that knowledge to identify questions, to acquire new knowledge, to explain scientific phenomena, and to draw evidence-based conclusions about science-related issues, understanding of the characteristic features of science as a form of human knowledge and enquiry, awareness of how science and technology shape our material, intellectual, and cultural environments, and willingness to engage in science-related issues, and with the ideas of science, as a reflective citizen.’

The domain consists of four interrelated aspects:

  • Context – life situations involving science and technology. Contexts are personal, social or global and may relate to health, natural resources, environment, hazard or the frontiers of science and technology.
  • Knowledge – knowledge of the natural world (covering physical systems, living systems, earth and space systems and technology systems) and knowledge about science itself (scientific enquiry and scientific explanations).
  • Competencies , of which  three are identified: identify scientific issues, explain phenomena scientifically and use scientific evidence.
  • Attitudes, including an interest in science, support for scientific enquiry and a motivation to act responsibly towards the natural world.

A 6-level proficiency scale is defined with the top levels explained as follows:

  • At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.
  • At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis.

.

Denham Sunset by Gifted Phoenix

Denham Sunset by Gifted Phoenix

.

 Changes in Average Performance in Reading, Maths and Science

The OECD published PISA outcomes for maths, science and reading on 3 December 2013.

Similarly, the PISA National Report on England, published simultaneously, covers the three core assessments.

This section looks briefly at the headline average scores and rankings across the selected sample of twelve jurisdictions, principally to enable comparisons to be drawn with the subsequent analysis of high achievers’ performance.

I apologise in advance for any transcription errors. Please let me know if you spot any and I will correct the tables accordingly.

.

Reading

Table 4 below gives the headline average numerical scores and ranks in reading from PISA 2006, 2009 and 2012 respectively.

.

Table 4

Country 2012 2009 2006
score rank score rank score rank
Australia 512↓ 13↓ 515↑ 9↓ 513 7
Canada 523↓ 8↓ 524↓ 6↓ 527 4
Finland 524↓ 6↓ 536↓ 3↓ 547 2
Hong Kong 545↑ 2↑ 533↓ 4↓ 536 3
Ireland 523↑ 7↑ 496↓ 21↓ 517 6
S Korea 536↓ 5↓ 539↓ 2↓ 556 1
New Zealand 512↓ 13↓ 521 7↓ 521 5
Shanghai 570↑ 1= 556 1 N/A N/A
Singapore 542↑ 3↑ 526 5 N/A N/A
Taiwan 523↑ 8↑ 495↓ 23↓ 496 16
UK (England) 500↑ 23↑ 495↓ 25↓ 496 17
US 498↓ 24↓ 500 17 N/A N/A
OECD Average 496↑ 493↓ 495

.

Shanghai has retained the ascendancy it established in 2009, adding a further 14 points to its average 2009 score. Whereas it was only 17 points beyond its nearest competitor in 2009, that lead has now been extended to 25 points.

South Korea’s performance has fallen slightly and it has been leapfrogged in the rankings by Hong Kong (up 12 points), Singapore (up 16 points), and Japan (not included in the table).

Two countries making even more significant improvements are Taiwan (up 28 points) and Ireland (up 27 points). Conversely, the performance of Finland (down 12 points) and New Zealand (down 9 points) has noticeably declined. Finland’s performance has been declining since 2006.

Results remain broadly unchanged in Australia, Canada, England, South Korea and the USA. South Korea has been unable to make up the ground it lost in 2009.

Ireland’s huge improvement from a very similar starting point in 2009 throws England’s lack of progress into sharper relief, although it is largely catching up lost ground in 2009, having performed relatively well in 2006.

England, like the US, continues to perform slightly above the OECD average, but has fallen further behind the Asian Tigers. The gap with the world’s leader in each assessment is now 70 points (up from 60 in 2006),

.

Maths

Table 5 below sets out scores and rankings in maths since PISA 2006

.

Table 5

Country 2012 2009 2006
  score rank score rank score rank
Australia 504↓ 19↓ 514↓ 15↓ 520 13
Canada 518↓ 13↓ 527= 10↓ 527 7
Finland 519↓ 12↓ 541↓ 6↓ 548 2
Hong Kong 561↑ 3= 555↑ 3 547 3
Ireland 501↑ 20↑ 487↓ 32↓ 501 22
S Korea 554↑ 5↓ 546↓ 4 547 4
New Zealand 500↓ 23↓ 519↓ 13↓ 522 11
Shanghai 613↑ 1= 600 1 N/A N/A
Singapore 573↑ 2= 562 2 N/A N?A
Taiwan 560↑ 4↑ 543↓ 5↓ 549 1
UK (England) 495↑ 25↑ 493↓ 27↓ 495 24
US 481↓ 36↓ 487↑ 31↑ 474 35
OECD Average 494↓   496↓   497  

 .

The overall picture is rather similar to that for reading.

Shanghai (up 13 points) and Singapore (up 11 points) continue to stretch away at the head of the field. Taiwan (up 17 points) has also made significant improvement and is now close behind Hong Kong.

There has been relatively more modest improvement in Hong Kong and South Korea (which has been overtaken by Taiwan).

Elsewhere, Ireland has again made significant headway and is back to the level it achieved in 2006. But Finland’s score has plummeted 22 points. New Zealand is not far behind (down 19). There have also been significant falls in the performance of Australia (down 10) Canada (down 9) and the US (down 6).

The US is now trailing 13 points below the OECD average, having failed to sustain the substantial improvement it made in 2009.

In England meanwhile, results are largely unchanged, though now just above the OECD average rather than just below it.

The gap between England and world leader Shanghai has reached 118 points, compared with a gap in 2006 between England and world leader Taiwan of 54 points. The gap between England and its main Commonwealth competitors has narrowed, but only as a consequence of the significant declines in the latter.

.

Science

Table 6 below provides the same data in respect of science.

.

Table 6

Country 2012 2009 2006
  score rank score rank score rank
Australia 521↓ 16↓ 527= 10↓ 527 8
Canada 525↓ 10↓ 529↓ 8↓ 534 3
Finland 545↓ 5↓ 554↓ 2↓ 563 1
Hong Kong 555↑ 2↑ 549↑ 3↓ 542 2
Ireland 522↑ 15↑ 508 20 508 20
S Korea 538= 7↓ 538↑ 6↑ 522 11
New Zealand 516↓ 18↓ 532↑ 7 530 7
Shanghai 580↑ 1= 575 1 N/A N/A
Singapore 551↑ 3↑ 542 4 N/A N/A
Taiwan 523↑ 13↓ 520↓ 12↓ 532 4
UK (England) 516↑ 18↓ 515↓ 16↓ 516 14
US 497↓ 28↓ 502↑ 23↑ 489 29
OECD Average 501=   501↑   498  

 .

Shanghai is again out in front, having repeated the clean sweep it achieved in 2009.

However, it has managed only a 5-point improvement, while Taiwan has improved by 13 points and Singapore by 9 points. Hong Kong has moved up by 6 points and Taiwan by 3 points, but South Korea’s score is unchanged from 2009.

New Zealand has dropped by 16 points and Finland by 9 points compared with 2009. There have been comparatively smaller declines in Australia and Canada, while Ireland has once again improved dramatically, by 14 points, and – in this case – the improvement is not simply clawing back ground lost in 2009.

England remains comfortably above the OECD average, but has made negligible improvement since 2006. US performance has dropped back below the OECD average as it has lost some of the ground it made up in 2009.

The gap between England and the world leaders is comparable with that in maths and significantly lower than in reading. The gap is now 64 points, compared with just 47 points in 2006.

.

Overall

Overall, the Asian Tigers have consolidated their positions by maintaining improvement in all three domains, though South Korea appears to be struggling to maintain the success of earlier years.

Finland and New Zealand are in worrying decline while Ireland is making rapid progress in the opposite direction.

.

.

The US results are stagnant, remaining comparatively poor, particularly in maths.

England has broadly maintained its existing performance profile, neither improving nor declining significantly. But, it is conspicuously losing ground on the world leaders, especially in maths. Other than in science it is close to the OECD average.

There is nothing here to give comfort to either the previous Government or the present incumbents. There might be some limited relief – even a degree of shadenfreude – in the fact that several better-placed nations are falling back more severely. But of course one cannot win the ‘global race’ by simply standing still.

.

Floral by Gifted Phoenix

Floral by Gifted Phoenix

 .

Changes in High Achievers’ Performance

So much for the average headline figures.

The remainder of this post is focused on  high achievement data. The ensuing sections once more examine reading, maths and science in that order, followed by a section on all-rounders.

.

Reading

Table 7 shows how the percentage achieving higher levels in reading has changed since PISA 2006, providing separate columns for Level 6 and above level 5 respectively (there was no Level 6 in 2006)..

.

Table 7

Country 2012 2009 2006
Level 6 Levels 5 and 6 Level 6 Levels 5+6 Level 5
Australia 1.9 11.7 2.1 12.8 10.6
Canada 2.1 12.9 1.8 12.8 14.5
Finland 2.2 13.5 1.6 14.5 16.7
Hong Kong 1.9 16.8 1.2 12.4 12.8
Ireland 1.3 11.4 0.7 7.0 11.7
S Korea 1.6 14.2 1.0 12.9 21.7
New Zealand 3.0 13.9 2.9 15.8 15.9
Shanghai 3.8 25.1 2.4 19.4 N/A
Singapore 5.0 21.2 2.6 15.7 N/A
Taiwan 1.4 11.8 0.4 5.2 4.7
UK (England) 1.3 9.1 1.0 8.1 9.2
US 1.0 7.9 1.5 9.9 N/A
OECD Average 1.1 8.4 1.0 7.0 8.6

 

This reveals that:

  • In 2012, Singapore has a clear lead on its competitors at Level 6, but it is overtaken by Shanghai at Level 5 and above. New Zealand also remains comparatively strong at Level 6, but falls back significantly when Levels 5 and 6 are combined.
  • The other Asian Tigers do not perform outstandingly well at Level 6: Hong Kong, South Korea and Taiwan are all below 2.0%, behind Canada and Finland. However, all but Taiwan outscore their competitors when Levels 5 and 6 are combined.
  • Hong Kong, Shanghai, Singapore and Taiwan are all making fairly strong progress over time. Patterns are rather less discernible for other countries, though there is a downward trend in the US.
  • In Finland, New Zealand and Canada – countries that seem to be falling back overall – the percentage of Level 6 readers continues to improve. This might suggest that the proportion of the highest performers in reading is not significantly affected when national performance begins to slide.
  • When judged against these world leaders, England’s comparative performance is brought into much clearer perspective. At Level 6 it is not far behind Taiwan, South Korea and even Hong Kong. But, at Level 5 and above, the gap is somewhat more pronounced. England is improving, but very slowly.
  • The comparison with Taiwan is particularly stark. In 2006, England had roughly twice as many students performing at Level 5. By 2009 Taiwan had caught up some of this ground and, by 2012, it had overtaken.

Table 8 compares changes since PISA 2006 in national performance at Level 5 and above with changes at Level 1 and below.

This is intended to reveal the balance between top and bottom – and whether this sample of world-leading and other English-speaking jurisdictions is making consistent progress at either end of the spectrum.

.

 Table 8

Country Levels 5 (and 6 from 2009) Level 1 (or equivalent) and below
2006 2009 2012 2006 2009 2012
Australia 10.6 12.8 11.7 13.4 14.3 14.2
Canada 14.5 12.8 12.9 11.0 10.3 10.9
Finland 16.7 14.5 13.5 4.8 8.1 11.3
Hong Kong 12.8 12.4 16.8 7.2 8.3 6.8
Ireland 11.7 7.0 11.4 12.2 17.2 9.7
S Korea 21.7 12.9 14.2 5.7 5.8 7.6
New Zealand 15.9 15.8 13.9 14.6 14.3 16.3
Shanghai N/A 19.4 25.1 N/A 4.1 2.9
Singapore N/A 15.7 21.2 N/A 12.4 9.9
Taiwan 4.7 5.2 11.8 14.3 15.6 11.5
UK (England) 9.2 8.1 9.1 18.9 18.4 16.7
US N/A 9.9 7.9 N/A 17.7 16.7
OECD Average 8.6 7.0 8.4 20.1 18.8 18

 

We can see that:

  • The countries with the highest proportion of students at Level 5 and above tend to have the lowest proportion at Level 1 and below. In Shanghai in 2012, there is a 22% percentage point gap between these two populations and fewer than 3 in every hundred fall into the lower attaining group.
  • Singapore is much closer to Shanghai at the top end than it is at the bottom. But even Shanghai seems to be making faster progress at the top than at the bottom, which might suggest that it is approaching the point at which the proportion of low achievers cannot be further reduced.
  • Compared with Hong Kong and South Korea, Singapore has a higher proportion of both high achievers and low achievers.
  • Whereas Taiwan had three times as many low achievers as high achievers in 2006, by 2012 the proportions were broadly similar, but progress at the top end is much faster than at the bottom.
  • The decline in Finland has less to do with performance at the top end (which has fallen by three percentage points) than with performance at the bottom (which has increased by more than six percentage points).
  • Canada has consistently maintained a higher percentage of high achievers than low achievers, but the reverse is true in Australia. In New Zealand the percentage at the top is declining and the percentage at the bottom is increasing. The gap between the two has narrowed slightly in England, but not significantly so.
  • To catch up with Shanghai, England has to close a gap of some 16 percentage points at the top end, compared with one of around 14 percentage points at the bottom.

The PISA National Report on England offers some additional analysis, noting that 18 jurisdictions had a higher proportion of pupils than England at Level 5 or above in 2012, including all those that outperformed England overall (with the exception of Estonia and Macao), and also France and Norway.

The National Report relies more heavily on comparing the performance of learners at the 5th and 95th percentiles in each country, arguing that:

‘This is a better measure for comparing countries than using the lowest and highest scoring pupils, as such a comparison may be affected by a small number of pupils in a country with unusually high or low scores.’

This is true in the sense that a minimum sample of 4,500 PISA participants would result in fewer than 100 at Level 6 in many jurisdictions.

On the other hand, the National Report fails to point out that analysis on this basis is not particularly informative about comparative achievement of the criterion-referenced standards denoted by the PISA thresholds.

It says rather more about the spread of performance in each country and rather less about direct international comparisons.

Key points include:

  • In England the score of learners at the 5th percentile was 328, compared with 652 at the 95th percentile. This difference of 324 points is slightly larger than the OECD average difference of 310 points. More than two-thirds of OECD countries had a smaller difference between these percentiles.
  • Compared with PISA 2012, the score of high achievers at the 95th percentile in PISA 2009 increased by six points to 652, while the score of low achievers at the 5th percentile fell by six points to 328. This increase in the attainment gap is higher than in 2009 (312) but lower than in 2006 (337). Thirteen OECD countries reported a wider spread of attainment than England.
  • Of countries outperforming England, only Japan (325 points), Singapore (329 points) Belgium (339 points) and New Zealand (347 points) demonstrated a similar or wider spread of attainment. Shanghai had the lowest difference (259 points) followed by Estonia (263).
  • The strongest performing jurisdictions at the 95th percentile were Singapore (698), Shanghai (690) and Japan (689), compared with 652 for England.
  • Amongst jurisdictions ranked higher than England, only the Netherlands, Liechtenstein, Estonia and Macao secured a lower score at the 95th percentile. Only Belgium reported a lower score at the 5th percentile.

.

Maths

Turning to maths, Table 9 illustrates changes in the pattern of high achievement since 2006, again showing the percentages performing at Level 6 and above Level 5 respectively.

.

Table 9

Country 2012 2009 2006
  Level 6 Levels 5 + 6 Level 6 Levels 5+6 Level 6 Levels 5+6
Australia 4.3 14.8 4.5 16.4 4.3 16.4
Canada 4.3 16.4 4.4 18.3 4.4 18
Finland 3.5 15.2 4.9 21.6 6.3 24.4
Hong Kong 12.3 33.4 10.8 30.7 9 27.7
Ireland 2.2 10.7 0.9 6.7 1.6 10.2
S Korea 12.1 30.9 7.8 25.5 9.1 27.1
New Zealand 4.5 15.0 5.3 18.9 5.7 18.9
Shanghai 30.8 55.4 26.6 50.7 N/A N/A
Singapore 19.0 40.0 15.6 35.6 N/A N/A
Taiwan 18.0 37.2 11.3 28.5 11.8 31.9
UK (England) 3.1 12.4 1.7 9.9 2.5 11.2
US 2.2 9.0 1.9 9.9 1.3 7.7
Average 3.3 12.6 3.1 12.7 3.3 13.4

.

The variations between countries tend to be far more pronounced than in reading:

  • There is a huge 28 percentage point spread in performance at Level 6 within this sample – from 2% to 30% – compared with a three percentage point spread in reading. The spread at Level 5 and above is also significantly larger – 46 percentage points compared with 17 percentage points in reading.
  • Shanghai has an 11 percentage point lead over its nearest competitor at Level 6 and an even larger 15 percentage point lead for Level 5 and above. Moreover it has improved significantly on both counts since 2009. Well over half its sample is now performing at Level 5 or above and almost a third are at Level 6.
  • Singapore and Taiwan are the next best performers, both relatively close together. Both are improving but, following a small dip in 2009, Taiwan is improving at a faster rate – faster even than Shanghai.
  • Hong Kong and South Korea also have similar 2012 profiles, as they did back in 2006. South Korea also lost ground in 2009, but is now improving at a faster rate than Hong Kong.
  • Finland appears to be experiencing quite significant decline: the proportion of Level 6 performers in 2012 is not far short of half what it was in 2006 and performance above Level 5 has fallen by more than nine percentage points. This is a somewhat different pattern to reading, in that the top performers are also suffering from the overall decline.

.

.

  • Australia, Canada and New Zealand have maintained broadly the same performance over time, though all are showing a slight falling off at Level 5 and above, and in New Zealand this also applies at Level 6.
  • After a serious slump in 2006, Ireland has overtaken its 2006 position. Meanwhile, the US has been making some progress at Level 6 but is less convincing at Level 5 and above.
  • Once again, this comparison does not particularly flatter England. It is not too far behind the Commonwealth countries and declining Finland at Level 6 but the gap is slightly larger at Level 5 and above. That said, England has consistently performed below the OECD average and remains in that position.
  • There are, however, some grounds for domestic celebration, in that England has improved by 2.5% at Level 5 and above, and by 1.4% at Level 6. This rate of improvement bears comparison with Hong Kong, albeit from a much lower base. It suggests a narrowing gap between England and its Commonwealth counterparts.

Table 10 gives the comparison with achievement at the bottom end of the distribution, setting out the percentages performing at different levels.

.

Table 10

Country Levels 5 and 6 Level 1 and below
  2006 2009 2012 2006 2009 2012
Australia 16.4 16.4 14.8 13.0 15.9 18.6
Canada 18 18.3 16.4 10.8 11.4 13.8
Finland 24.4 21.6 15.2 5.9 7.8 12.2
Hong Kong 27.7 30.7 33.4 9.5 8.8 8.5
Ireland 10.2 6.7 10.7 16.4 20.9 16.9
S Korea 27.1 25.5 30.9 8.8 8.1 9.1
New Zealand 18.9 18.9 15.0 14.0 15.5 22.6
Shanghai N/A 50.7 55.4 N/A 4.8 3.7
Singapore N/A 35.6 40.0 N/A 9.8 8.3
Taiwan 31.9 28.5 37.2 11.9 12.8 12.8
UK (England) 11.2 9.9 12.4 19.9 19.8 21.7
US 7.7 9.9 9.0 28.1 23.4 25.9
Average 13.4 12.7 12.6 21.3 22.0 23.0

.

Key points include:

  • The same pattern is discernible amongst the strongest performers as was evident with reading: those with the highest percentages at the top end tend to have the lowest percentages at the bottom. If anything this distinction is even more pronounced. Shanghai records a 52 percentage point gap between its highest and lowest performers and the latter group is only slightly larger than the comparable group in the reading assessment.
  • Amongst the Asian Tigers, the ratio between top and bottom is at least 3:1 in favour of the top. For most of the other countries in the sample, there is never more than a 7 percentage point gap between top and bottom, but this stretches to 9 in the case of England and 13 for the USA. Needless to say, the low achievers are in the majority in both cases.
  • Although the percentages for top and bottom in Australia are broadly comparable, it has shifted since 2006 from a position where the top end was in the majority by 3 percentage points to almost a mirror image of that pattern. In New Zealand, the lower achievers have increased by almost 9 percentage points, almost double the rate of decline at the top end, as their ‘long tail’ grows significantly longer.
  • Apart from Shanghai, only Singapore, Hong Kong and South Korea have fewer than 10% in the lower performing category. Despite its reputation as a meritocratic environment, Singapore gets much closer to Shanghai at the bottom of the distribution than it does at the top. The same is true of Hong Kong and South Korea.
  • It is also noticeable that none of the Tigers is making extraordinary progress at the bottom end. Hong Kong has reduced this population by 1% since 2003, Singapore by 1.5% since 2006, Shanghai by only 0.9% since 2006. The percentage has increased in South Korea and Taiwan. Improvement has been significantly stronger at the top of the distribution. Again this might suggest that the Tigers are closing in on the point where they cannot improve further at the bottom end.
  • In Finland, the percentage achieving the higher levels has fallen by over 9 percentage points since 2006, while the increase at the lower levels is over 6 percentage points. This compares with a 3 point fall at the top and a 6 point rise at the bottom in reading. The slump amongst Finland’s high achievers is clearly more pronounced in maths.
  • England’s 9.3 percentage point gap between the top and bottom groups in 2012 is lightly larger than the 8.7 point gap in 2006. It has a whopping 43 percentage point gap to make up on Shanghai at the top end, and an 18 point gap at the bottom. England is just on the right side of the OECD average at the bottom and just on the wrong side at the top.

.

.

The National Report notes that all jurisdictions ahead of England in the rankings had a higher percentage of learners at Level 5 or above.

As for percentiles

  • The difference between the 5th percentile (335 points) and the 95th percentile (652 points) was 316 in England. The average difference for OECD countries was 301, only slightly lower than that.
  • Ten countries had a greater difference than this, five of them amongst those the highest overall mean scores. Others were Israel, Belgium, Slovakia, New Zealand and France.
  • Whereas the difference between the lowest and highest percentiles has increased very slightly across all OECD countries, this is more pronounced in England, increasing from 285 points in 2009 to 316 points in 2012. This is attributable to decreasing scores at the 5th percentile (350 in 2006, 349 in 2009 and 335 in 2012) compared with changes at the 95th percentile (643 in 2006, 634 in 2009 and 652 in 2012).

.

Science

Table 11 compares the performance of this sample of PISA participants at the higher levels in the science assessment on the last three occasions.

.

Table 11

Country 2012 2009 2006
  Level 6 Levels 5 + 6 Level 6 Levels 5+6 Level 6 Levels 5+6
Australia 2.6 13.5 3.1 14.6 2.8 14.6
Canada 1.8 11.3 1.6 12.1 2.4 14.4
Finland 3.2 17.1 3.3 18.7 3.9 20.9
Hong Kong 1.8 16.7 2 16.2 2.1 15.9
Ireland 1.5 10.8 1.2 8.7 1.1 9.4
S Korea 1.1 11.7 1.1 11.6 1.1 10.3
New Zealand 2.7 13.4 3.6 17.6 4 17.6
Shanghai 4.2 27.2 3.9 24.3 N/A N/A
Singapore 5.8 22.7 4.6 19.9 N/A N/A
Taiwan 0.6 8.4 0.8 8.8 1.7 14.6
UK (England) 1.9 11.7 1.9 11.6 3.0 14.0
US 1.1 7.4 1.3 9.2 1.5 9.1
Average 1.2 8.4 1.1 8.5 1.3 8.8

.

In science, the pattern of high achievement has more in common with reading than maths. It shows that:

  • There is again a relatively narrow spread of performance between this sample of jurisdictions – approaching five percentage points at Level 6 and 20 percentage points at Level 5 and above.
  • As in reading, Singapore outscores Shanghai at the top level 6, but is outperformed by Shanghai at Level 5 and above. Both are showing steady improvement, but Singapore’s improvement at Level 6 is more pronounced than Shanghai’s.
  • Finland remains the third best performer, although the proportion of learners achieving at both Level 6 and Level 5 plus has been declining slightly since 2006.
  • Another similarity with reading is that Australia, Finland and New Zealand all perform significantly better at Level 6 than Hong Kong, South Korea and Taiwan. Hong Kong alone performs equally well at Level 5 and above. None of these three Asian Tigers has made significant progress since 2006.
  • In Australia, Canada, New Zealand and the US there has also been relatively little progress over time – indeed some evidence to suggest a slight decline. Conversely, Ireland seems to be moving forward again after a slight dip at Level 5 and above in 2009.

.

.

  • England was a strong performer in 2006, broadly comparable with many of its competitors. But it fell back significantly in 2009 and has made no progress since then. The proportions are holding up but there is no substantive improvement since 2009, unlike in maths and (to a lesser extent) reading. However England continues to perform somewhat higher than the OECD average. There is an interesting parallel with Taiwan, although that country dipped even further than England in 2009.

Table 12 provides the comparison with the proportions achieving the lower thresholds.

.

Table 12

Country Levels 5 and 6 Levels 1 and Below
  2006 2009 2012 2006 2009 2012
Australia 14.6 14.6 13.5 12.8 12.6 13.6
Canada 14.4 12.1 11.3 10.0 9.5 10.4
Finland 20.9 18.7 17.1 4.1 6.0 7.7
Hong Kong 15.9 16.2 16.7 8.7 6.6 5.6
Ireland 9.4 8.7 10.8 15.5 15.1 11.1
S Korea 10.3 11.6 11.7 11.2 6.3 6.7
New Zealand 17.6 17.6 13.4 13.7 13.4 16.3
Shanghai N/A 24.3 27.2 N/A 3.2 2.7
Singapore N/A 19.9 22.7 N/A 11.5 9.6
Taiwan 14.6 8.8 8.4 11.6 11.1 9.8
UK (England) 14.0 11.6 11.7 16.7 14.8 14.9
US 9.1 9.2 7.4 24.4 18.1 18.2
Average 8.8 8.5 8.4 19.3 18.0 17.8

 .

  • Amongst the top performers the familiar pattern reappears. In 2012 Shanghai has 27% in the top categories against 2.7% in the bottom categories. This is very similar to reading (25.1% against 2.9%). At the bottom end, Shanghai’s nearest competitors are Hong Kong and South Korea, while Singapore and Taiwan are each approaching 10% at these levels. This is another similarity with reading (whereas, in maths, Singapore is more competitive at the lower end).
  • Since 2009, Shanghai has managed only a comparatively modest 0.5% reduction in the proportion of its students at the bottom end, compared with an increase of almost 3% at the top end. This may lend further support to the hypothesis that it is approaching the point at which further bottom end improvement is impossible.
  • No country has made consistently strong progress at the bottom end, though Ireland has made a significant improvement since 2009. There has been steady if unspectacular improvement in Hong Kong, Taiwan and Singapore. South Korea, having achieved a major improvement in 2009 has found itself unable to continue this positive trend.
  • Finland’s negative trend is consistent since 2006 at both ends of the achievement spectrum, though the decline is not nearly as pronounced as in maths. In science Finland is maintaining a ratio of 2:1 in favour of the performers at the top end, while percentages at top and bottom are now much closer together in both reading and maths.
  • There are broadly similar negative trends at top and bottom alike in the Commonwealth countries of Australia, Canada and New Zealand, although they have fallen back in fits and starts. In New Zealand the balance between top and bottom has shifted from being 4 percentage points in favour of the top end in 2006, to 3 percentage points in favour of the bottom end by 2012.
  • A similar gap in favour of lower achievers also exists in England and is unchanged from 2009. By comparison with the US (which is a virtual mirror image of the top-bottom balance in Finland, Singapore or South Korea) it is in a reasonable position, rather similar to New Zealand, now that it has fallen back.
  • England has a 1.5 percentage point gap to make up on Shanghai at the top end of the distribution, compared with a 12.2 percentage point gap at the bottom.

The PISA 2012 National Study reports that only the handful of jurisdictions shown in Table 11 above has a larger percentage of learners achieving Level 6. Conversely, England has a relatively large number of low achievers compared with these jurisdictions.

Rather tenuously, it argues on this basis that:

‘Raising the attainment of lower achievers would be an important step towards improving England’s performance and narrowing the gap between highest and lowest performers.’

When it comes to comparison of the 5th and 95th percentiles:

  • The score at the 5th percentile (343) and at the 95th percentile (674) gives a difference of 331 points, larger than the OECD average of 304 points. Only eight jurisdictions had a wider distribution: Israel, New Zealand, Luxembourg, Slovakia, Belgium, Singapore and Bulgaria.
  • The OECD average difference between the 5th and 95th percentiles has reduced slightly (from 311 in 2006 to 304 in 2012) and there has also been relatively little change in England.

.

Top-Performing All-Rounders

Volume 1 of the OECD’s ‘PISA 2012 Results’ document provides additional data about all-round top performers achieving Level 5 or above in each of the three domains.

.

PISA 2012 top performers Capture.

The diagram shows that 4.4% of learners across OECD countries achieve this feat.

This is up 0.3% on the PISA 2009 figure revealed in this PISA in Focus publication.

Performance on this measure in 2012, compared with 2009, amongst the sample of twelve jurisdictions is shown in the following Table 13. (NB that the UK figure is for the UK combined, not just England).

.

Table 13

2012 2009
%age rank %age rank
Australia 7.6 7 8.1 6
Canada 6.5 9 6.8 8
Finland 7.4 8 8.5 4
Hong Kong 10.9 4 8.4 5
Ireland 5.7 15 3.2 23
S Korea 8.1 5 7.2 7
New Zealand 8.0 6 9.9 3
Shanghai 19.6 1 14.6 1
Singapore 16.4 2 12.3 2
Taiwan 6.1 10 3.9 17
UK 5.7 15 4.6 14
US 4.7 18 5.2 11
Average 4.4 4.1

 .

In terms of percentage increases, the fastest progress on this measure is being made by Hong Kong, Ireland, Shanghai, Singapore and Taiwan. Shanghai has improved a full five percentage points and one in five of its students now achieve this benchmark.

The UK is making decent progress, particularly compared with Australia, Canada, Finland New Zealand and the US, which are moving in the opposite direction.

The Report notes:

‘Among countries with similar mean scores in PISA, there are remarkable differences in the percentage of top-performing students. For example, Denmark has a mean score of 500 points in mathematics in PISA 2012 and 10% of students perform at high proficiency levels in mathematics, which is less than the average of around 13%. New Zealand has a similar mean mathematics score of 500 points, but 15% of its students attain the highest levels of proficiency, which is above the average…these results could signal the absence of a highly educated talent pool for the future.

Having a large proportion of top performers in one subject is no guarantee of having a large proportion of top performers in the others. For example, Switzerland has one of the 10 largest shares of top performers in mathematics, but only a slightly-above-average share of top performers in reading and science.

Across the three subjects and across all countries, girls are as likely to be top performers as boys. On average across OECD countries, 4.6% of girls and 4.3% of boys are top performers in all three subjects…To increase the share of top-performing students, countries and economies need to look at the barriers posed by social background…the relationship between performance and students’… and schools’ organisation, resources and learning environment.’ (p65)

.

Denizen by Gifted Phoenix

Denizen by Gifted Phoenix

 

Conclusions

Priorities for Different Countries

On the basis of this evidence, it is possible to draw up a profile of the performance of different countries across the three assessments at these higher levels, and so make a judgement about the prospects in each of ‘a highly educated talent pool for the future’. The twelve jurisdictions in our sample might be advised as follows:

  • Shanghai should be focused on establishing ascendancy at Level 6 in reading and science, particularly if there is substance to the suspicion that scope for improvement at the bottom of the spectrum is now rather limited. Certainly it is likely to be easier to effect further improvement at the very top.
  • Singapore has some ground to catch up with Shanghai at Level 6 in maths. It has narrowed that gap by three percentage points since 2009, but there is still some way to go. Otherwise it should concentrate on strengthening its position above Level 5, where Shanghai is also conspicuously stronger.
  • Hong Kong needs to focus on Level 6 in reading and science, but perhaps also in maths where it has been extensively outpaced by Taiwan since 2009. At levels 5 and above it faces strong pressure to maintain proximity with Shanghai and Singapore, as well as marking the charge made by Taiwan in reading and maths. Progress in science is relatively slow.
  • South Korea should also pay attention to Level 6 in reading and science. It is improving faster than Hong Kong at Level 6 in maths but is also losing ground on Taiwan. That said, although South Korea now seems back on track at Level 5 and above in maths, but progress remains comparatively slow in reading and science, so both Levels 5 and 6 need attention.
  • Taiwan has strong improvement in reading and maths since 2009, but is deteriorating in science at both Levels 5 and 6. It still has much ground to pick up at Level 6 in reading. Its profile is not wildly out of kilter with Hong Kong and South Korea.
  • Finland is bucking a downward trend at Level 6 in reading and slipping only slightly in science, so the more noticeable decline is in maths. However, the ground lost is proportionately greater at Level 5 and above, once again more prominently in maths. As Finland fights to stem a decline at the lower achievement levels, it must take care not to neglect those at the top.
  • Australia seems to be slipping back at both Levels 5 and 6 across all three assessments, while also struggling at the bottom end. There are no particularly glaring weaknesses, but it needs to raise its game across the board.
  • Canada is just about holding its own at Level 6, but performance is sliding back at Level 5 and above across all three domains. This coincides with relatively little improvement and some falling back at the lower end of the achievement distribution. It faces a similar challenge to Finland’s although not so pronounced.
  • New Zealand can point to few bright points in an otherwise gloomy picture, one of which is that Level 6 performance is holding up in reading. Elsewhere, there is little to celebrate in terms of high achievers’ performance. New Zealand is another country that, in tackling more serious problems with the ‘long tail’, should not take its eye off the ball at the top.

.

.

  • The US is also doing comparatively well in reading at Level 6, but is otherwise either treading water or slipping back a little. Both Level 6 and Level 5 and above need attention. The gap between it and the world’s leading countries continues to increase, suggesting that it faces future ‘talent pool’ issues unless it can turn round its performance.
  • Ireland is a good news story, at the top end as much as the bottom. It has caught up lost ground and is beginning to push beyond where it was in 2006. Given Ireland’s proximity, the home countries might want to understand more clearly why their nearest neighbour is improving at a significantly faster rate. That said, Ireland has significant room for improvement at both Level 6 and Level 5 and above.
  • England’s performance at Level 6 and Level 5 and above has held up surprisingly well compared with 2009, especially in maths. When the comparison is solely historical, there might appear to be no real issue. But many other countries are improving at a much stronger rate and so England (as well as the other home countries) risks being left behind in the ‘global race’ declared by its Prime Minister. The world leaders now manage three times as many Level 6 performers in science, four times as many in reading and ten times as many in maths. It must withstand the siren voices urging it to focus disproportionately at the bottom end.

.

Addressing These Priorities

It is far more straightforward to pinpoint these different profiles and priorities than to recommend convincingly how they should be addressed.

The present UK Government believes firmly that its existing policy direction will deliver the improvements that will significantly strengthen its international competiveness, as judged by PISA outcomes. It argues that it has learned these lessons from careful study of the world’s leading performers and is applying them carefully and rigorously, with due attention to national needs and circumstances.

.

.

But – the argument continues – it is too soon to see the benefits of its reforms in PISA 2012, such is the extended lag time involved in improving the educational outcomes of 15 year-olds. According to this logic, the next Government will reap the significant benefits of the present Government’s reform programme, as revealed by PISA 2015.

Recent history suggests that this prediction must be grounded more in hope than expectation, not least because establishing causation between indirect policy interventions and improved test performance must surely be the weakest link in the PISA methodology.

But, playing devil’s advocate for a moment, we might reasonably conclude that any bright spots in England’s performance are attributable to interventions that the previous Government got right between five and ten years ago. It would not be unreasonable to suggest that the respectable progress made at the top PISA benchmarks is at least partly attributable to the national investment in gifted education during that period.

We might extend this argument by suggesting a similar relationship between progress in several of the Asian Tigers at these higher levels and their parallel investment in gifted education. Previous posts have drawn attention to the major programmes that continue to thrive in Hong Kong, Singapore, South Korea and Taiwan.

Shanghai might have reached the point where success in mainstream education renders investment in gifted education unnecessary. On the other hand, such a programme might help it to push forward at the top in reading and science – perhaps the only conspicuous chink in its armour. There are lessons to be learned from Singapore. (Gifted education is by no means dormant on the Chinese Mainland and there are influential voices pressing the national government to introduce more substantive reforms.)

Countries like Finland might also give serious consideration to more substantive investment in gifted education geared to strengthening high attainment in these core domains. There is increasingly evidence that the Finns need to rethink their approach.

.

.

The relationship between international comparisons studies like PISA and national investment in gifted education remains poorly researched and poorly understood, particularly how national programmes can most effectively be aligned with and support such assessments.

The global gifted education community might derive some much-needed purpose and direction by establishing an international study group to investigate this issue, providing concrete advice and support to governments with an interest.

.

GP

December 2013

A Summer of Love for English Gifted Education? Episode 3: Improving Fair Access to Oxbridge

.

This post is a critical examination of policy and progress on improving progression for the highest attainers from disadvantaged backgrounds to selective universities, especially Oxford and Cambridge.

.

.

It:

  • Uncovers evidence of shaky statistical interpretation by these universities and their representative body;
  • Identifies problems with the current light-touch regulatory and monitoring apparatus, including shortcomings in the publication of data and reporting of progress at national level;
  • Proposes a series of additional steps to address this long-standing shortcoming of our education system.

.

Background

summer of love 1967 by 0 fairy 0

summer of love 1967 by 0 fairy 0

Regular readers may recall that I have completed two parts of a trilogy of posts carrying the optimistic strapline ‘A Summer of Love for Gifted Education’.

The idea was to structure these posts around three key government publications.

  • This final part was supposed to analyse another DfE-commissioned research report, an ‘Investigation of school- and college- level strategies to raise the Aspirations of High-Achieving Disadvantaged Pupils to pursue higher education’.

We know from the published contract (see attachment in ‘Documents’ section) that this latter study was undertaken by TNS/BMRB and the Institute for Policy Studies in Education (IPSE) based at London Metropolitan University. The final signed off report should have been produced by 28 June 2013 and published within 12 weeks of approval, so by the end of September. As I write, it has still to appear, which would suggest that there is a problem with the quality and/or size of the evidence base.

In the five months since the appearance of Part Two I have published a series of posts developing the themes explored in the first two-thirds of my incomplete trilogy.

But what to do about the missing final episode of ‘A Summer of Love’, which was going to develop this latter fair access theme in more detail?

My initial idea was to survey and synthesise the large number of other recently published studies on the topic. But, as I reviewed the content of these publications, it struck me that such a post would be stuffed full of descriptive detail but lack any real bite – by which I mean substantial and serious engagement with the central problem.

I decided to cut to the chase.

I also decided to foreground material about the highest reaches of A level attainment and progression to Oxbridge, not because I see the issue solely in these stratospheric terms, but because:

  • The top end of fair access is important in its own right, especially for those with a gifted education perspective. Oxford and Cambridge consistently declare themselves a special case and I wanted to explore the substance of their position.
  • There is compelling evidence that Oxford and Cambridge are amongst the weakest performers when it comes to fair access for the highest attaining disadvantaged learners. There are reasons why the task may be comparatively more difficult for them but, equally, as our most prestigious universities, they should be at the forefront when it comes to developing and implementing effective strategies to tackle the problem.
  • The Government has itself made Oxbridge performance a litmus test of progress (or lack of progress) on fair access and on higher education’s wider contribution to social mobility.

The first part of the post briefly reviews the range of measures and regulatory apparatus devoted to improving fair access. This is to provide a frame from which to explore the available data and its shortcomings, rather than an in-depth analysis of relative strengths and weaknesses. Readers who are familiar with this background may prefer to skip it.

The mid-section concentrates on the limited data in the public domain and how it has been (mis)interpreted.

The final section reviews the criticisms made by the SMCPC and, while endorsing them thoroughly, offers a set of further proposals – many of them data-driven – for ratcheting up our collective national efforts to reverse the unsatisfactory progress made to date.

.

A Troubling Tale of Unnecessary Complexity and Weak Regulation

.

A Proliferation of Measures

There is little doubt that we have a problem in England when it comes to progression to selective, competitive higher education (however defined) by learners from disadvantaged backgrounds (however defined).

We may not be unique in that respect, but that does not alter the fact that the problem is longstanding and largely unresolved.

The recent ‘State of the Nation 2013’ Report from the SMPCPC says ‘there has been little change in the social profile of the most elite institutions for over a decade’, adding that ‘while some of the building blocks are in place to lift children off the bottom, opening up elites remains elusive.’

Part of the problem is that the debates about these respective definitions continue to receive disproportionate coverage. Such debates are sometimes deployed as a diversionary tactic, intentionally drawing us away from the unpalatable evidence that we are making decidedly poor headway in tackling the core issue.

The definitional complexities are such that they lend themselves to exploitation by those with a vested interest in preserving the status quo and defending themselves against what they regard as unwonted state intervention.

I shall resist the temptation to explore the comparative advantages and disadvantages of different measures, since that would risk falling into the trap I have just identified.

But I do need to introduce some of the more prominent – and pin down some subtle distinctions – if only for the benefit of readers in other countries.

One typically encounters four different categorisations of competitive, selective higher education here in the UK:

  • Oxbridge – a convenient shorthand reference to Oxford and Cambridge Universities. These two institutions are commonly understood to be qualitatively superior to other UK universities and, although that advantage does not apply universally, to every undergraduate course and subject, there is some academic support for treating them as a category in their own right.
  • Russell Group – The Russell Group was formed in 1994 and originally comprised 17 members. There are currently 24 members, 20 of them located in England, including Oxford and Cambridge. Four institutions – Durham, Exeter, Queen Mary’s and York – joined as recently as 2012 and membership is likely to increase as the parallel 1994 Group has just disbanded. DfE (as opposed to BIS) often uses Russell Group membership as its preferred proxy for selective, competitive higher education, although there are no objective criteria that apply exclusively to all members.
  • Sutton Trust 30The Sutton Trust originally identified a list of 13 universities, derived from ‘average newspaper league table rankings’. This list – Birmingham, Bristol, Cambridge, Durham, Edinburgh, Imperial, LSE, Nottingham, Oxford, St Andrews, UCL, Warwick and York – still appears occasionally in research commissioned by the Trust, although it was subsequently expanded to 30 institutions. In ‘Degrees of Success’, a July 2011 publication, they were described thus:

‘The Sutton Trust 30 grouping of highly selective universities comprises universities in Scotland, England and Wales with over 500 undergraduate entrants each year, where it was estimated that less than 10 per cent of places are attainable to pupils with 200 UCAS tariff points (equivalent to two D grades and a C grade at A-level) or less. These 30 universities also emerge as the 30 most selective according to the latest Times University Guide.’

The full list includes all but two of the Russell Group (Queen Mary’s and Queen’s Belfast) plus eight additional institutions.

‘The HEIs included in this group change every year; although 94% of HEIs remained in the top third for 5 consecutive years, from 2006/07 to 2010/11. The calculation is restricted to the top three A level attainment; pupils who study other qualifications at Key Stage 5 will be excluded. Institutions with a considerable proportion of entrants who studied a combination of A levels and other qualifications may appear to have low scores. As the analysis covers students from schools and colleges in England, some institutions in other UK countries have scores based on small numbers of students. As this measure uses matched data, all figures should be treated as estimates.’

This categorisation includes seven further mainstream universities (Aston, City, Dundee, East Anglia, Goldsmiths, Loughborough, Sussex) plus a range of specialist institutions.

Indicators of educational disadvantage are legion, but these are amongst the most frequently encountered:

  • Eligibility for free school meals (FSM): DfE’s preferred measure. The term is misleading since the measure only includes learners who meet the FSM eligibility criteria and for whom a claim is made, so eligibility of itself is insufficient. Free school meals are available for learners in state-funded secondary schools, including those in sixth forms. From September 2014, eligibility will be extended to all in Years R, 1 and 2 and to disadvantaged learners in further education and sixth form colleges. The phased introduction of Universal Credit will also impact on the eligibility criteria (children of families receiving Universal Credit between April 2013 and March 2014 are eligible for FSM, but the cost of extending FSM to all Universal Credit recipients once fully rolled out is likely to be prohibitive). We do not yet know whether these reforms will cause DfE to select an alternative preferred measure and, if so, what that will be. Eligibility for the Pupil Premium is one option, more liberal than FSM, though this currently applies only to age 16.
  • Residual Household Income below £16,000: This is broadly the income at which eligibility for free school meals becomes available. It is used by selective universities (Oxford included) because it can be applied universally, regardless of educational setting and whether or not free school meals have been claimed. Oxford explains that:

‘Residual income is based on gross household income (before tax and National Insurance) minus certain allowable deductions. These can include pension payments, which are eligible for certain specified tax relief, and allowances for other dependent children.’

The threshold is determined through the assessment conducted by Student Finance England, so is fully consistent with its guidance.

  • Low participation schools: This measure focuses on participation by school attended rather than where students live. It may be generic – perhaps derived from the Government’s experimental destinations statistics – or based on admissions records for a particular institution. As far as I can establish, there is no standard or recommended methodology: institutions decide for themselves the criteria they wish to apply.
  • POLAR (Participation Of Local Areas): HEFCE’s area-based classification of participation in higher education. Wards are categorised in five quintiles, with Quintile 1 denoting those with lowest participation. The current edition is POLAR 3.
  • Other geodemographic classifications: these include commercially developed systems such as ACORN and MOSAIC based on postcodes and Output Area Classification (OAC) based on census data. One might also include under this heading the Indices of Multiple Deprivation (IMD) and the associated sub-domain Income Deprivation Affecting Children Index (IDACI).
  • National Statistics Socio-Economic Classification (NS-SEC): an occupationally-based definition of socio-economic status applied via individuals to their households. There are typically eight classes:
  1. Higher managerial, administrative and professional
  2. Lower managerial, administrative and professional
  3. Intermediate
  4. Small employers and own account workers
  5. Lower supervisory and technical
  6. Semi routine
  7. Routine
  8. Never worked and long term unemployed

Data is often reported for NS-SEC 4-7.

Sitting alongside these measures of disadvantage is a slightly different animal – recruitment from state-funded schools and colleges compared with recruitment from the independent sector.

While this may be a useful social mobility indicator, it is a poor proxy for fair access.

Many learners attending independent schools are from comparatively disadvantaged backgrounds, and of course substantively more learners at state-maintained schools are comparatively advantaged.

The Office For Fair Access (OFFA) confirms that:

‘in most circumstances we would not approve an access agreement allowing an institution to measure the diversity of its student body solely on the basis of the numbers of state school pupils it recruits….it is conceivable that a university could improve its proportion of state school students without recruiting greater proportions of students from disadvantaged groups.’

Nevertheless, independent/state balance continues to features prominently in some access agreements drawn up by selective universities and approved by OFFA.

There is a risk that some institutions are permitted to give this indicator disproportionate attention, at the expense of their wider commitment to fair access.

 .

Securing National Improvement

Given the embarrassment of riches set out above, comparing progress between institutions is well-nigh impossible, let alone assessing the cumulative impact on fair access at national level.

When it came to determining their current strategy, the government of the day must have faced a choice between:

  • Imposing a standard set of measures on all institutions, ignoring complaints that those selected were inappropriate for some settings, particularly those that were somehow atypical;
  • Allowing institutions to choose their own measures, even though that had a negative impact on the rate of improvement against the Government’s own preferred national indicators; and
  •  A half-way house which insisted on universal adoption of one or two key measures while permitting institutions to choose from a menu of additional measures, so creating a basket more or less appropriate to their circumstances.

For reasons that are not entirely clear – but presumably owe something to vigorous lobbying from higher education interests – the weaker middle option was preferred and remains in place to this day.

The standard-setting and monitoring process is currently driven by OFFA, though we expect imminently the final version of a National Strategy for Access and Student Success, developed jointly with HEFCE.

A new joint process for overseeing OFFA’s access agreements (from 2015/16) and HEFCE’s widening participation strategic statements (from 2014-2017) will be introduced in early 2014.

There were tantalising suggestions that the status quo might be adjusted through work on the wider issue of evaluation.

An early letter referred to plans to:

‘Commission feasibility study to establish if possible to develop common evaluation measures that all institutions could adopt to assess the targeting and impact of their access and student success work’.

The report would be completed by Spring 2013.

Then an Interim Report on the Strategy said the study would be commissioned in ‘early 2013 to report in May 2013’ (Annex B).

It added:

‘Informal discussions with a range of institutional representatives have indicated that many institutions would welcome a much clearer indication of the kind of evidence and indicators that we would wish to see. Therefore a key strand within the strategy development will be work undertaken with the sector to develop an evaluation framework to guide them in their efforts to evidence the impact of their activity. Within this, we intend to test the feasibility of developing some common measures for the gathering of high-level evidence that might be aggregated to provide a national picture. We will also investigate what more can be done by national bodies including ourselves to make better use of national data sets in supporting institutions as they track the impact of their interventions on individual students.’

However, HEFCE’s webpage setting out research and stakeholder engagement in support of the National Strategy still says the study is ‘to be commissioned’ and that the publication date is ‘to be confirmed’.

I can find no explanation of the reasons for this delay.

For the time being, OFFA is solely responsible for issuing guidance to institutions on the content of their access agreements, approving the Agreements and monitoring progress against them.

OFFA’s website says:

‘Universities and colleges set their own targets based on where they need to improve and what their particular institution is trying to achieve under its access agreement…These targets must be agreed by OFFA. We require universities and colleges to set themselves at least one target around broadening their entrant pool. We also encourage (but do not require) them to set themselves further targets, particularly around their work on outreach and, where appropriate, retention. Most choose to do so. We normally expect universities and colleges to have a range of targets in order to measure their progress effectively. When considering whether targets are sufficiently ambitious, we consider whether they represent a balanced view of the institution’s performance, and whether they address areas where indicators suggest that the institution has furthest to go to improve access.

From 2012-13, in line with Ministerial guidance, we are placing a greater emphasis on progress against targets. We would not, however, impose a sanction solely on the basis of a university or college not meeting its targets or milestones.’

The interim report on a National Strategy suggests that – informally at least – many universities recognise that this degree of flexibility is not helpful to their prospects of improving fair access, either individually or collectively.

But the fact that the promised work has not been undertaken might imply a counterforce pushing in precisely the opposite direction.

The expectations placed on universities are further complicated by the rather unclear status of the annual performance indicators for widening participation of under-represented groups supplied by the Higher Education Statistics Agency (HESA).

HESA’s table for young full-time first degree entrants shows progress by each HEI against benchmarks for ‘from state schools or colleges’, ‘from NS-SEC classes 4, 5, 6 and 7’ and ‘from low participation neighbourhoods (based on POLAR3 methodology)’ respectively.

HESA describes its benchmarks thus:

‘Because there are such differences between institutions, the average values for the whole of the higher education sector are not necessarily helpful when comparing HEIs. A sector average has therefore been calculated which is then adjusted for each institution to take into account some of the factors which contribute to the differences between them. The factors allowed for are subject of study, qualifications on entry and age on entry (young or mature).’

HESA’s benchmarks are clearly influential in terms of the measures adopted in many access agreements (and much of the attention given to the state versus independent sector intake may be attributable to them).

On the other hand, the indicators receive rather cavalier treatment in the most recent access agreements from Oxford and Cambridge. Oxford applies the old POLAR2 methodology in place of the latest POLAR3, while Cambridge adjusts the POLAR3 benchmarks to reflect its own research.

The most recent 2011/12 HESA results for Oxford and Cambridge are as follows:

.

Institution       State schools        NS-SEC 4-7     LPN (POLAR3)
Benchmark Performance Benchmark Performance Benchmark Performance
Oxford 71.2% 57.7% 15.9% 11.0% 4.7% 3.1%
Cambridge 71.4% 57.9% 15.9% 10.3% 4.5% 2.5%

.

That probably explains why Oxbridge would prefer an alternative approach! But the reference to further work in the Interim Strategy perhaps also suggests that few see these benchmarks as the best way forward.

.

National Targets

The Government also appears in something of a bind with its preferred measures for monitoring national progress.

When it comes to fair access (as opposed to widening participation) the Social Mobility Indicators rely exclusively on the gap between state and independent school participation at the most selective HEIs, as defined by BIS.

As noted above, this has major shortcomings as a measure of fair access, though more validity as a social mobility measure.

The relevant indicator shows that the gap held between 37% and 39% between 2006 and 2010, but this has just been updated to reflect an unfortunate increase to 40% in 2010/11.

BIS uses the same measure as a Departmental Performance Indicator for its work on higher education.  The attachment on the relevant gov.uk page is currently the wrong one – which might indicate that BIS is less than comfortable with its lack of progress against the measure.

DfE takes a different approach declaring an ‘Outcome of Education’ indicator:

‘Outcome of education:

i)             Percentage of children on free school meals progressing to Oxford or Cambridge*.

ii)            Percentage of children on free school meals progressing to a Russell Group university*.

iii)           Percentage of children on free school meals progressing to any university*.

iv)           Participation in education and work based training at age 16 to 17

*Available June 2013’

But progress against this indicator is nowhere to be found in the relevant section of the DfE website or, as far I can establish, anywhere within the DfE pages on gov.uk.

.

.

Oxbridge Access Agreement Targets for 2014/15

Perhaps the best way to link this section with the next is by showing how Oxford and Cambridge have decided to frame the targets in their access agreements for 2014/15

Oxford has OFFA’s agreement to target:

  • Schools and colleges that secure limited progression to Oxford. They use ‘historic UCAS data’ to estimate that ‘in any one year up to 1,680…will have no students who achieve AAA grades but, over a three-year period they may produce a maximum of two AAA candidates’. They also prioritise an estimated 1,175 institutions which have larger numbers achieving AAA grades ‘but where the success rate for an application to Oxford is below 10%’. In 2010, 19.4% of Oxford admissions were from these two groups and it plans to increase the proportion to 25% by 2016-17;
  • UK undergraduates from disadvantaged socio-economic backgrounds, based on ‘ACORN postcodes 4 and 5’. Some 7.6% of admissions came from these postcodes in 2010/11 and Oxford proposes to reach 9.0% by 2016/17.
  • UK undergraduates from neighbourhoods with low participation in higher education, as revealed by POLAR2. It will focus on ‘students domiciled in POLAR quintiles 1 and 2’. In 2012, 10.6% are from this group and Oxford proposes to increase this to 13.0% by 2016-17.

In addition to a target for admitting disabled students, Oxford also says it will monitor and report on the state/independent school mix, despite evidence ‘that this measure is often misleading as an indicator of social diversity’. It notes that:

‘30% of 2012 entrants in receipt of the full Oxford Bursary (students with a household income of £16,000 or less) were educated in the independent sector…The University will continue to monitor the level of students from households with incomes of £16,000 or less. It is considered that these are the most financially disadvantaged in society, and it is below this threshold that some qualify for receipt of free schools meals, and the pupil premium. The University does not consider that identifying simply those students who have been in receipt of free school meals provides a suitably robust indicator of disadvantage as they are not available in every school or college with post-16 provision, nor does every eligible student choose to receive them.

There are no national statistics currently available on the number of students whose household income is £16,000 or less and who attain the required academic threshold to make a competitive application to Oxford. In 2011-12, around one in ten of the University’s UK undergraduate intake was admitted from a household with this level of declared income.’

Meanwhile, Cambridge proposes only two relevant targets, one of them focused on the independent/state divide:

  • Increase the proportion of UK resident students admitted from UK state sector schools and colleges to between 61% and 63%. This is underpinned by the University’s research finding that ‘the proportion of students nationally educated at state schools securing examination grades in subject combinations that reflect our entrance requirements and the achievement level of students admitted to Cambridge stands at around 62%’.
  • Increase the proportion of UK resident students from low participation neighbourhoods to approximately 4% by 2016. It argues:

‘Currently HESA performance indicators and other national datasets relating to socio-economic background do not take adequate account of the entry requirements of individual institutions. Whilst they take some account of attainment, they do not do so in sufficient detail for highly selective institutions such as Cambridge where the average candidate admitted has 2.5 A* grades with specific subject entry requirements. For the present we have adjusted our HESA low participation neighbourhood benchmark in line with the results of our research in relation to state school entry and will use this as our target.’

Each of these approaches has good and bad points. Cambridge’s is more susceptible to the criticism that it is overly narrow. There is no real basis to compare the relative performance of the two institutions since there is negligible overlap between their preferred indicators. That may be more comfortable for them, but it is not in the best interests of their customers, or of those seeking to improve their performance.

 

Investigating the Data on High Attainment and Fair Access to Oxbridge

Those seeking statistics about high attainment amongst disadvantaged young people and their subsequent progression to Oxbridge are bound to be disappointed.

There is no real appreciation of the excellence gap in this country and this looks set to continue. The fact that gaps between advantaged and disadvantaged learners are typically wider at the top end of the attainment distribution seems to have acted as a brake on the publication of data that proves the point.

It is possible that the current round of accountability reforms will alter this state of affairs, but this has not yet been confirmed.

For the time being at least, almost all published statistics about high A level attainment amongst disadvantaged learners have come via answers to Parliamentary Questions. This material invariably measures disadvantage in terms of FSM eligibility.

Information about the admission of disadvantaged learners to Oxbridge is equally scant, but a picture of sorts can be built up from a mixture of PQ replies, university admission statistics and the DfE’s destination measures. The material supplied by the universities draws on measures other than FSM.

The following two sections set out what little we know, including the ever important statistical caveats.

.

High Attainment Data

  • In 2003, 94 students (1.9%) eligible for FSM achieved three or more A grades at A level. The figures relate to 16-18 year-olds in maintained schools only who were eligible for FSM at age 16. They do not include students in FE sector colleges (including sixth from colleges) who were previously eligible for FSM. Only students who entered at least one A level, applied A level or double award qualification are included. (Parliamentary Question, 6 April 2010, Hansard (Col 1346W))
  • In 2008, 160 students (3.5%) eligible for FSM achieved that outcome. The figures relate to 16-18 year-olds, in maintained schools only, who were eligible for FSM at age 16. They do not include students in FE sector colleges (including sixth from colleges) who were previously eligible for FSM. Only students who entered at least one A level, applied A level or double award qualification are included. (Parliamentary Question, 6 April 2010, Hansard (Col 1346W))
  • In 2008/09, 232 pupils at maintained mainstream schools eligible for FSM achieved three or more A grades at A level (including applied A level and double award), 179 of them attending comprehensive schools. The figures exclude students in FE and sixth form colleges previously eligible for FSM. (Parliamentary Question, 7 April 2010, Hansard (Col 1451W))
  • The number of Year 13 A level candidates eligible for FSM in Year 11 achieving 3 or more A grade levels (including applied A levels and double award) were: 2006 – 377; 2007 – 433; 2008 – 432; 2009 – 509. These figures include students in both the schools and FE sectors.(Parliamentary Question, 27 July 2010, Hansard (Col 1223W))

 .

To summarise, the total number of students who were FSM-eligible at age 16 and went on to achieve three or more GCE A levels at Grade A*/A – including those in maintained schools, sixth form and FE colleges – has been increasing significantly since 2006.

2006 2007 2008 2009 2010 2011
Number 377 433 432 509 ? 546

The overall increase between 2006 and 2011 is about 45%.

 .

Oxbridge Admission/Acceptance Data

  • The number of learners eligible for and claiming FSM at age 15 who progressed to Oxford or Cambridge by age 19 between 2005 and 2008 (rounded to the nearest five) were:
2005/06 2006/07 2007/08 2008/09
Oxford 25 20 20 25
Cambridge 20 25 20 20
TOTAL 45 45 40 45

Sources: Parliamentary Question, 13 December 2010, Hansard (Col 549W) and Parliamentary Question 21 February 2012, Hansard (Col 755W)

.

[Postscript (January 2014):

In January 2014, BIS answered a further PQ which provided equivalent figures for 2009/10 and 2010/11 – again rounded to the nearest five and derived from matching the National Pupil Database (NPD), HESA Student Record and the Individualised Learner Record (ILR) owned by the Skills Funding Agency.

The revised table is as follows:

  2005/06 2006/07 2007/08 2008/09 2009/10 2010/11
Oxford 25 20 20 25 15 15
Cambridge 20 25 20 20 25 25
TOTAL 45 45 40 45 40 40

 

Sources:

Parliamentary Question, 13 December 2010, Hansard (Col 549W)

Parliamentary Question 21 February 2012, Hansard (Col 755W)

Parliamentary Question 7 January 2014, Hansard (Col 191W)

Although the 2010/11 total is marginally more positive than the comparable figure derived from the Destination Indicators (see below) this confirms negligible change overall during the last six years for which data is available.  The slight improvement at Cambridge during the last two years of the sequence is matched by a corresponding decline at Oxford, from what is already a desperately low base.]

.

Number %age FSM Number FSM
UK HEIs 164,620 6% 10,080
Top third of HEIs 49,030 4% 2,000
Russell Group 28,620 3% 920
Oxbridge 2,290 1% 30

.

.

These are experimental statistics and all figures – including the 30 at Oxbridge – are rounded to the nearest 10. The introductory commentary explains that:

‘This statistical first release (experimental statistics) on destination measures shows the percentage of students progressing to further learning in a school, further education or sixth-form college, apprenticeship, higher education institution or moving into employment or training.’

It adds that:

‘To be included in the measure, young people have to show sustained participation in an education or employment destination in all of the first 2 terms of the year after they completed KS4 or took A level or other level 3 qualifications. The first 2 terms are defined as October to March.’

The Technical Notes published alongside the data also reveal that: It includes only learners aged 16-18 and those who have entered at least one A level or an equivalent L3 qualification;  the data collection process incorporates ‘an estimate of young people who have been accepted through the UCAS system for entry into the following academic year’ but ‘deferred acceptances are not reported as a distinct destination’; and FSM data for KS5 learners relates to those eligible for and claiming FSM in Year 11.

  • Cambridge’s 2012 intake ‘included 50+ students who had previously been in receipt of FSM’ (It is not stated whether all were eligible in Year 11, so it is most likely that this is the number of students who had received FSM at one time or another in their school careers.) This shows that Cambridge at least is collecting FSM data that it does not publish amongst its own admission statistics or use in its access agreement. (Cambridge University Statement, 26 September 2013)
  • In 2012, Cambridge had 418 applications from the most disadvantaged POLAR2 quintile (4.6% of all applicants) and, of those, 93 were accepted (3.6% of all acceptances) giving a 22.2% success rate.(Cambridge University Admission Statistics 2012 (page 23))

.

To summarise, the numbers of disadvantaged learners progressing to Oxbridge are very small; exceedingly so as far as those formerly receiving FSM are concerned.

Even allowing for methodological variations, the balance of evidence suggests that, at best, the numbers of FSM learners progressing to Oxbridge have remained broadly the same since 2005.

During that period, the concerted efforts of the system described above have had zero impact. The large sums invested in outreach and bursaries have made not one iota of difference.

This is true even though the proportion achieving the AAA A level benchmark has increased by about 45%. If Oxbridge admission was solely dependent on attainment, one would have expected a commensurate increase, to around 65 FSM entrants per year.

On the basis of the 2010/11 Destination Indicators, we can estimate that, whereas Oxbridge admits approximately 8% of all Russell Group students, it only admits slightly over 3% of Russell Group FSM students. If Oxbridge achieved the performance of its Russell Group peers, the numbers of formerly FSM admissions would be over 100 per year.

.

Misleading Use of This Data

To add insult to injury, this data is frequently misinterpreted and misused. Here are some examples, all of which draw selectively on the data set out above.

  • Of the 80,000 FSM-eligible students in the UK only 176 received three As at A level…more than one quarter of those students….ended up at either Oxford or Cambridge – Nicholson (Oxford Undergraduate Admissions Director, Letter to Guardian, 7 March 2011)
  • ‘Of the 80,000 children eligible for free school meals in the UK in 2007, only 176 received 3 As at A level. Of those 45 (more than a quarter) got places at Oxford or Cambridge’ (Undated Parliamentary Briefing ‘Access and admissions to Oxford University’ )
  • ‘The root causes of underrepresentation of students from poorer backgrounds at leading universities include underachievement in schools and a lack of good advice on subject choices. For example, in 2009 only 232 students who had been on free school meals (FSM) achieved 3As at A-level or the equivalent.  This was 4.1% of the total number of FSM students taking A-levels, and less than an estimated 0.3% of all those who had received free school meals when aged 15.’ (Russell Group Press release, 23 July 2013).
  • ‘Such data as is available suggests that less than 200 students per year who are recorded as being eligible for FSM secure grades of AAA or better at A level. The typical entrance requirement for Cambridge is A*AA, and so on that basis the University admits in excess of one quarter of all FSM students who attain the grades that would make them eligible for entry.’ (Cambridge University Statement, 26 September 2013)
  • ‘According to data produced by the Department for Children, Schools and Families, of the 4,516 FSM students who secured a pass grade at A Level in 2008 only 160 secured the grades then required for entry to the University of Cambridge (ie AAA). Students who were eligible for FSM therefore make up less than 1% of the highest achieving students nationally each year.

Assuming that all 160 of these students applied to Oxford or Cambridge in equal numbers (ie 80 students per institution) and 22 were successful in securing places at Cambridge (in line with the 2006-08 average) then this would represent a success rate of 27.5% – higher than the average success rate for all students applying to the University (25.6% over the last three years). In reality of course not every AAA student chooses to apply to Oxford or Cambridge, for instance because neither university offers the course they want to study, e.g. Dentistry.’ (Cambridge Briefing, January 2011 repeated in Cambridge University Statement, 26 September 2013)

.

.

To summarise, Oxford, Cambridge and the Russell Group are all guilty of implying that FSM-eligible learners in the schools sector are the only FSM-eligible learners progressing to selective universities.

They persist in using the school sector figures even though combined figures for the school and FE sectors have been available since 2010.

Oxbridge’s own admission statistics show that, in 2012:

  • 9.6% of acceptances at Cambridge (332 students) were extended to students attending sixth form, FE and tertiary colleges (UK figures)
  • 10.5% of UK domiciled acceptances at Oxford (283 students) were extended to students attending sixth form colleges and FE institutions of all types

We can rework Cambridge’s calculation using the figure of 546 students with three or more A*/A grades in 2011:

  • assuming that all applied to Oxford and Cambridge in equal numbers gives a figure of 273 per institution
  • assuming a success rate of 25.6% – the average over the last three years
  • the number of FSM students that would have been admitted to Cambridge is roughly 70.

Part of the reason high-attaining disadvantaged students do not apply to Oxbridge may be because they want to study the relatively few mainstream subjects, such as dentistry, which are not available.

But it is highly likely that other factors are at play, including the perception that Oxbridge is not doing all that it might to increase numbers of disadvantaged students from the state sector.

If this favourable trend in A level performance stalls, as a consequence of recent A level reforms, it will not be reasonable – in the light of the evidence presented above – for Oxbridge to argue that this is impacting negatively on the admission of FSM-eligible learners.

.

Building on the work of the SMCPC

 

‘Higher Education: The Fair Access Challenge’

There is no shortage of publications on fair access and related issues. In the last year alone, these have included:

Easily the most impressive has been the Social Mobility and Child Poverty Commission’s ‘Higher Education: The Fair Access Challenge’ (June 2013), though it does tend to rely a little too heavily on evidence of the imbalance between state and independent-educated students.

.

.

It examines the response of universities to recommendations first advanced in an earlier publication ‘University Challenge: How Higher Education Can Advance Social Mobility’ (2012) published by Alan Milburn, now Chair of the Commission, in his former role as Independent Reviewer on Social Mobility.

The analysis sets out key points from the earlier work:

  • Participation levels at the most selective universities by the least advantaged are unchanged since the mid-90s.
  • The most advantaged young people are seven times more likely to attend the most selective universities than the most disadvantaged.
  • The probability of a state secondary pupil eligible for FSM in Year 11 entering Oxbridge by 19 is almost 2000 to 1; for a privately educated pupil the probability is 20 to 1.

New research is presented to show that the intake of Russell Group universities has become less socially representative in the last few years:

  • The number of state school pupils entering Russell Group Universities has increased by an estimated 2.6% from 2002/03 to 2011/12, but the commensurate increase in privately educated entrants is 7.9%. The proportion of young full-time state-educated entrants has consequently fallen from 75.6% to 74.6% over this period. The worst performers on this measure are: Durham (-9.2%), Newcastle (-4.6%), Warwick (-4.5%) and Bristol (-3.9%). The best are: Edinburgh (+4.6%), UCL (+3.3%), LSE (+3.0%) and Southampton (2.9%). The Oxbridge figures are: Cambridge (+0.3%) and Oxford (+2.3%).
  • Similarly, the proportion of young full-time entrants from NS-SEC classes 4-7 has fallen from 19.9% in 2002/03 to 19.0% in 2011/12. A table (reproduced below) shows that the worst offenders on this measure are Queen’s Belfast (-4.6%), Liverpool (-3.2%), Cardiff (-2.9%) and Queen Mary’s (-2.7%). Conversely, the best performers are Nottingham (+2.2%), York (+0.9%), Warwick and LSE (+0.8%). The figures for Oxbridge are: Cambridge (-1.0%) and Oxford (0.0%).

.

NC-SEC Capture.

  • An estimated 3,700 state-educated learners have the necessary grades for admission to Russell Group universities but do not take up places. This calculation is based on the fact that, if all of the 20 Russell Group universities in England achieved their HESA widening participation benchmarks, they would have recruited an extra 3,662 students from state schools. (The benchmarks show how socially representative each intake would be if it were representative of all entrants with the grades required for entry – though see Cambridge’s reservations on this point, above.) Some universities would need to increase significantly the percentage of state students recruited – for example, Bristol and Durham (26.9%), Oxford (23.4%) and Cambridge (23.3%).
  • Using the same methodology to calculate the shortfall per university in NS-SEC 4-7 students results in the table below, showing the worst offenders to require percentage increases of 54.4% (Cambridge), 48.5% (Bristol), 45.5% (Oxford) and 42,2% (Durham). Conversely, Queen Mary’s, Queen’s Belfast, LSE and Kings College are over-recruiting from this population on this measure.

.

NS sec Capture 2.

  • Even if every Russell Group university met the self-imposed targets in its access agreement, the number of ‘missing’ state educated students would drop by only 25% by 2016/17, because the targets are insufficiently ambitious. (This is largely because only seven have provided such targets in their 2013/14 access agreements and there are, of course, no collective targets.)
  • Boliver’s research is cited to show that there is a gap in applications from state school pupils compared with those educated in the independent sector. But there is also evidence that a state school applicant needs, on average, one grade higher in their A levels (eg AAA rather than AAB) to be as likely to be admitted as an otherwise identical student from the independent sector.
  • A Financial Times analysis of 2011 applications to Oxford from those with very good GCSEs found that those from independent schools were 74% more likely to apply than those from the most disadvantaged state secondary schools. Amongst applicants, independently educated students were more than three times as likely to be admitted as their peers in disadvantaged state schools. They were also 20% more likely to be admitted than those at the 10% most advantaged state secondary schools. As shown by the table below, the probabilities involved varied considerably. The bottom line is that the total probability of a place at Oxford for an independent school student is 2.93%, whereas the comparable figure for a student at one of the 10% most disadvantaged state secondary schools is just 0.07%.

.

NS sec Capture 3

When it comes to the causes of the fair access gap, subject to controls for prior attainment, the report itemises several contributory factors, noting the limited evidence available to establish their relative importance and interaction:

  • low aspirations among students, parents and teachers
  • less knowledge of the applications process, problems in demonstrating potential through the admissions process and a tendency to apply to the most over-subscribed courses
  • not choosing the right  A-level subjects and teachers’ under-prediction of expected A level grades
  • a sense that selective universities ‘are socially exclusive and “not for the likes of them”’

The Report states unequivocally that:

‘The Social Mobility and Child Poverty Commission is deeply concerned about the lack of progress on fair access. The most selective universities need to be doing far more to ensure that they are recruiting from the widest possible pool of talent. The Commission will be looking for evidence of a step change in both intention and action in the years to come.’

It identifies several areas for further action, summarising universities’ responses to ‘University Challenge’:

  • Building links between universities and schools: The earlier report offered several recommendations, including that universities should have explicit objectives to help schools close attainment gaps. No evidence is given to suggest that such action is widespread, though many universities are strengthening their outreach activities and building stronger relationships with the schools sector. Several universities highlighted the difficulties inherent in co-ordinating their outreach activity given the demise of Aimhigher, but several retain involvement in a regional partnership.
  • Setting targets for fair access: The earlier report recommended that HE representative bodies should set statistical targets for progress on fair access over the next five years. This was not met positively:

‘Representative bodies in the Higher Education Sector did not feel this would be a useful step for them to take, saying that it was difficult to aggregate the different targets that individual institutions set themselves. There was also a feeling among some highly selective institutions that the report overestimated the number of students who have the potential to succeed at the most selective universities.’

Nevertheless, the Commission is insistent:

The Commission believes it is essential that the Russell Group signals its determination to make a real difference to outcomes by setting a clear collective statistical target for how much progress its members are aiming to make in closing the ‘fair access gap’. Not doing so risks a lack of sustained focus among the most selective universities’.

  • Using contextual admissions data: The report argues that ‘there is now a clear evidence base that supports the use of contextual data’. Recommendations from the earlier report were intended to universalise the use of contextual data, including commitment from the various representative bodies through a common statement of support and a collaborative guide to best practice. There is no sign of the former, although the Commission reports ‘widespread agreement that the use of contextual data during the admissions process should be mainstreamed’. However it notes that there is much more still to do. (The subsequent SPA publication should have helped to push forward this agenda.)
  • Reforming the National Scholarship Programme: The earlier report called on the Government to undertake a ‘strategic review of government funding for access’ to include the national Scholarship Programme (NSP). The suggestion that the imminent HEFCE/OFFA National Strategy should tackle the issue has been superseded by a Government decision to refocus the NSP on postgraduate education.
  • Postgraduate funding reform: The earlier report recommended work on a postgraduate loan scheme and further data collection to inform future decisions. The current report says that:

‘…the Government appears to have decided against commissioning an independent report looking at the issue of postgraduate access. This is very disappointing.’

and calls on it ‘to take heed’. However, this has again been superseded by the NSP announcement.

The SMCPC’s ‘State of the Nation 2013’ report reinforces its earlier publication, arguing that:

‘…despite progress, too much outreach work that aims to make access to university fairer and participation wider continues to rely on unproven methods or on work that is ad hoc, uncoordinated and duplicative… These are all issues that the higher education sector needs to address with greater intentionality if progress is to be made on breaking the link between social origin and university education.

The UK Government also needs to raise its game… much more needs to be done… to address the loss of coordination capacity in outreach work following the abolition of Aimhigher.’

It recommends that:

‘All Russell Group universities should agree five-year aims to close the fair access gap, all universities should adopt contextual admissions processes and evidence-based outreach programmes, and the Government should focus attention on increasing university applications from mature and part-time students.’

 .

What Else Might Be Done?

I set myself the challenge of drawing up a reform programme that would build on the SMCPC’s recommendations but would also foreground the key issues I have highlighted above, namely:

  • A significant improvement in the rate of progression for disadvantaged high-attaining learners to Oxbridge;
  • A more rigorous approach to defining, applying and monitoring improvement measures; and
  • The publication of more substantive and recent data

A determined administration that is prepared to take on the vested interests could do worse than pursue the following 10-point plan

  • 1. Develop a new approach to specifying universities’ fair access targets for young full-time undergraduate students. This would require all institutions meeting the BIS ‘most selective HEI’ criteria to pursue two universal measures and no more than two measures of their own devising, so creating a basket of no more than four measures. Independent versus state representation could be addressed as one of the two additional measures.
  • 2. The universal measures should relate explicitly to students achieving a specified A level threshold that has currency at these most selective HEIs. It could be pitched at the equivalent of ABB at A level, for example. The measures should comprise:
    • A progression measure for all learners eligible for the Pupil Premium in Year 11 of their secondary education (so a broader measure than FSM eligibility); and
    • A progression measure for all learners – whether or not formerly eligible for the Pupil Premium – attending a state-funded sixth form or college with a relatively poor historical record of securing places for their learners at such HEIs. This measure would be nationally defined and standardised across all institutions other than Oxbridge.
  • 3. In the case of Oxford and Cambridge the relevant A level tariff would be set higher, say at the equivalent of AAA grades at A level, and the nationally defined  ‘relatively poor historical record’ would reflect only Oxbridge admission.
  • 4. These two universal measures would be imposed on institutions through the new National Strategy for Access and Student Success. All institutions would be required to set challenging but realistic annual targets. There would be substantial financial incentives for institutions achieving their targets and significant financial penalties for institutions that fail to achieve them.
  • 5. The two universal measures would be embedded in the national Social Mobility Indicators and the KPIs of BIS and DfE respectively.
  • 6. Central Government would publish annually data setting out:
    • The number and percentage of formerly Pupil Premium-eligible learners achieving the specified A level thresholds for selective universities and Oxbridge respectively.
    • A ‘league table’ of the schools and colleges with relatively poor progression to selective universities and Oxbridge respectively.
    • A ‘league table’ of the universities with relatively poor records of recruitment from these schools and colleges.
    • A time series showing the numbers of students and percentage of their intake drawn from these two populations by selective universities and Oxbridge respectively each year. This should cover both applications and admissions.
  • 7. All parties would agree new protocols for data sharing and transparency, including tracking learners through unique identifiers across the boundaries between school and post-16 and school/college and higher education, so ensuring that the timelag in the publication of this data is minimal.
  • 8. Universities defend fiercely their right to determine their own undergraduate admissions without interference from the centre, meaning that the business of driving national improvement is much more difficult than it should be. But, given the signal lack of progress at the top end of the attainment distribution, there are strong grounds for common agreement to override this autonomy in the special case of high-achieving disadvantaged students.  A new National Scholarship Scheme should be introduced to support learners formerly in receipt of the Pupil Premium who go on to achieve the Oxbridge A Level tariff:
    • Oxford and Cambridge should set aside 5% additional places per year (ie on top of their existing complement) reserved exclusively for such students. On the basis of 2012 admissions figures, this would amount to almost exactly 250 places for England divided approximately equally between the two institutions (the scheme could be for England only or UK-wide). This would provide sufficient places for approximately 45% of those FSM learners currently achieving 3+ A*/A grades.
    • All eligible students with predicted grades at or above the tariff would be eligible to apply for one of these scholarship places. Admission decisions would be for the relevant university except that – should the full allocation not be taken up by those deemed suitable for admission who go on to achieve the requisite grades – the balance would be made available to the next best applicants until the quota of places at each university is filled.
    • The Government would pay a premium fee set 50% above the going rate (so £4,500 per student per annum currently) for each National Scholarship student admitted to Oxbridge. However, the relevant University would be penalised the full fee plus the premium (so £13,500 per student per year) should the student fail to complete their undergraduate degree with a 2.2 or better. Penalties would be offset against the costs of running the scheme. Assuming fees remain unchanged and 100% of students graduate with a 2.2 or better, this would cost the Government £1.125m pa.
  • 9. In addition, the Government would support the establishment of a National Framework Programme covering Years 9-13, along the lines set out in my November 2010 post on this topic with the explicit aim of increasing the number of Pupil Premium-eligible learners who achieve these tariffs. The budget could be drawn in broadly equal proportions from Pupil Premium/16-19 bursary, a matched topslice from universities’ outreach expenditure and a matched sum from the Government. If the programme supported 2,500 learners a year to the tune of £2,500 per year, the total steady state cost would be slightly over £30m, approximately £10m of which would be new money (though even this could be topsliced from the overall Pupil Premium budget).
  • 10. The impact of this plan would be carefully monitored and evaluated, and adjusted as appropriate to maximise the likelihood of success. It would be a condition of funding that all selective universities would continue to comply with the plan.

Do I honestly believe anything of this kind will ever happen?

.

flying pig capture

.

GP

November 2013

How High Attainers Feature in Ofsted Inspection and School Performance Tables (and what to do about it) – Part Two

.

This is the second and final part of a post about how the school accountability system reflects the performance of high attaining learners.

Part One considered recent amendments to Ofsted’s inspection guidance and compared Ofsted’s approach with how high attainers are defined in the School Performance Tables. It reviewed expectations of progression by high attainers and proposed that these should be increased.

Part Two:

  • Reviews how the next School Performance Tables (2013 results) will feature high attainers, compared with the current Tables (2012 results).
  • Explains how high attainers feature in the proposals for assessment and accountability reform set out in three recent consultation documents – covering primary, secondary and post-16 education respectively. This takes in the recently published Government response to the secondary consultation.
  • Offers guidance for schools on how they might set about planning to improve the performance of their high attainers, given current accountability arrangements and future prospects and
  • Proposes for further discussion a basket of key indicators that schools might publish and pursue alongside learners, parents and other stakeholders.

I have adopted a simple taxonomy of measures throughout the discussion that follows.

This distinguishes measures relating specifically to high attainers according to whether they feature:

  • Attainment: the achievement of specified grades or levels in assessments conducted at the end of a key stage of education.
  • Progress: the expected trajectory between two or more of these assessments, consistent with achieving commensurate outcomes in each.
  • Destination: the nature of the educational or other setting to which learners have moved at the end of a key stage.
  • Closing the Gap: the difference between the outcomes of disadvantaged and other learners on any of these measures, whether this is falling and, if so, by how much.
  • Comparison: how the performance of schools/colleges on any of these measures compares with broadly similar institutions.

I have also flagged up related measures of high attainment – and measures which reflect high attainment – where these have been applied to the entire cohort rather than separately to high attainers.

.

High Attainers in the 2012 and 2013 Performance Tables

We begin with a comparison between the 2012 and 2013 School Performance Tables covering the primary, secondary and 16-18 tables respectively.

The details of reporting in the 2013 Tables are drawn from the published Statement of Intent. This confirms that they should be published according to the standard timetable, in mid-December 2013 (primary) and late January 2014 (secondary and post-16) respectively, with any data not then ready for publication added as and when it becomes available.

For ease of reference I have included in brackets the national figure for each high attainer measure in the 2012 Tables (state-funded schools only).

 .

Primary Tables

Significant changes will be apparent in 2013. These are consequential on: the introduction of Level 4B+ and Level 6 performance; the removal of an overall level in English; the introduction of the grammar, punctuation and spelling (GPS) test and the addition of three year averages for specified measures.

Level 4B+ has been introduced as a new marker of ‘secondary readiness’. The reason given is that analysis of sub levels showed that, in 2012, only 47% of those with 4C in both English and maths went on to achieve 5 A*-C grade GCSEs including English and maths, while the comparable percentages for 4B and 4A were 72% and 81% respectively.

I pause only to note that, if the threshold is raised in this manner, there is even stronger logic behind the idea of raising the threshold for high attainers in parallel – an idea I floated in Part One of this post.

The table below compares the 2012 and 2013 measures using the taxonomy set out above.

 .

  2012 2013
High Attainers measures
Attainment % achieving L3- (0%), L4+ (99%), L5+ in both English and maths (72%) % achieving L3- and L5+, in all three of reading and maths tests and writing TA
KS1-2 value added measure (English and maths) (99.8) % achieving L4+ and L4B+ in reading test
% achieving L4+ in writing TA
% achieving L3-, L4+, L4B+, L5+ and L6 in GPS test
% achieving L3-, L4+, L4B+, L5+ and L6 in maths test
KS1-2 VA (reading, writing and maths)
Progress % making at least expected progress in English (87%) and maths (92%) % making at least  expected progress in each of reading, writing, maths
Destinations None None
Closing the gap None None
Comparison % achieving L4+ in English and maths (99%) % achieving L4+ in  reading, writing and maths
 
High Attainment measures
  % achieving L5+ in reading and maths tests and writing TA (20%) % achieving L5+ and L6 in reading test
  % achieving L5+ in English (37%) in maths (39%) and in reading (48%) % achieving L5+ and L6 in writing TA
  % achieving L5+ in English TA (36%), maths TA (40%), science TA (36%), reading TA (46%) and writing TA (28%) % achieving L5+ and L6 in English, reading, maths, science TA
  Average point score (28.2) Average point score (reading, writing, maths)

.

The number of attainment measures applied specifically to high attainers has increased, but it is not entirely clear why so many different combinations of levels will be reported for different elements (there are four variants in all). The extent of L6 attainment is clearly a factor, but cannot be the sole reason.

For the first time we will be able to compare the performance of disadvantaged and other learners for both L5 and L6 in GPS and maths, at least in theory, since very few schools are likely to have sufficient L6 performance by disadvantaged learners to register on this measure.

But this is a high attainment measure. We still cannot see what proportion of high attainers are disadvantaged – and how their performance compares with their peers.

A national primary destinations measure is not really feasible and would tell us little, unless receiving secondary schools are categorised on the basis of their performance in the secondary tables or their Ofsted rating. It might be interesting to see what proportion of high attainers (particularly disadvantaged high attainers) transferrto secondary schools deemed outstanding – compared with middle and low attainers – but the benefits would be limited. However, this might be more relevant at local level.

The basis of the comparison with other schools is important to understand, since this methodology will be carried forward in the accountability reforms reviewed below.

The probability is calculated of pupils achieving KS2 L4+ in both English and maths, based on the achievement of pupils with the same prior attainment at KS1. These probabilities are averaged to provide a figure for the school. Then a similar school group is established by selecting the 62 schools with immediately stronger average performance and the 62 with immediately weaker average performance, giving a group of 125 schools. An average Level per pupil is calculated from the average points score.

The same methodology is used in the secondary tables. Here the calculation is built on the probability of achieving 5+ A*-C grades in GCSE, or equivalent, plus English and maths GCSEs, and based on the achievement of pupils with the same prior attainment at KS2. In this case the similar school group is derived from the 27 schools immediately above and the 27 immediately below, giving a group of 55 comparator schools.

 .

Secondary Tables

Compared with the extensive changes to the Primary Tables in 2013, there is very little different in the Secondary Tables, apart from the introduction of an average grade per pupil (best 8) measure.

 .

2012 2013
High attainers measures
Attainment % achieving: 5+ A*-C GCSEs or equivalent including GCSEs in English and maths (94%) Ditto
% achieving Grades A*-C in GCSE English and maths (94.3%) Ditto
APS (best 8) all qualifications (398.5) and GCSE only (375.4) Ditto
Average grade per qualification and per GCSE Ditto
Average entries per pupil for all qualifications (12.4) and GCSE (9.7); Ditto
% entered for all EBacc subjects (46.3%);            % achieving all EBacc qualifications (38.5%); Ditto
EBacc VA in English (1000.2), maths (1000.1), science (1000.4), humanities (1000.8) and languages (1000.2) Ditto
Average grade per pupil (best 8)
Progress % making at least expected progress in English (96.6%) and maths (96.8%) Ditto
  VA (best 8) (1000.8) Ditto
Destinations None (see below) Published data included
Closing the Gap None Ditto
Comparisons % achieving 5+ A*-C GCSEs or equivalent, including GCSEs in English and maths (94%) Ditto
High Attainment measures None Ditto

 .

I have not included in the high attainment section the few additional attainment measures not  applied to high attainers (percentage achieving 5+ GCSEs at A*-C or equivalent (83%); percentage entered for and achieving EBacc subjects – English, maths, science, humanities and languages). These are not specifically high attainment measures and tell us relatively little.

As in the Primary Tables, there is a no analysis of schools’ success in closing the gap for high attainers. And, since there are no substantive high attainment measures in the Secondary Tables (with the possible exception of the EBacc), this omission is comparatively more significant.

Whereas we can see some L5+ and L6 performance for disadvantaged learners in the primary tables, there is no equivalent focus – say on 5+ GCSEs at A*/A including English and maths – in the secondary tables.

Destinations measures have already been published separately for 2010/11 and education destinations will be included in the 2013 Performance Tables, but the breakdown of destinations data by pupil characteristics does not include a prior attainment category.

(Compare this with the more cautious statements about the longer term use of KS4 destinations data set out below. The Statement of Intent is comparatively more cautious about the KS5 data.)

We have also had the separate announcement that Performance Tables will henceforward record only a learner’s first entry for any given GCSE examination (or IGCSE or Level 1/Level 2 Certificate).

This is too late for the imminent 2013 Tables, but will impact on the 2014 Tables (published in January 2015), when it will bite on all EBacc subjects. From 2015 it will impact on all subjects.

There are two schools of thought about the potential impact on high attainers.

One might argue that this change should not affect them unduly, since they are much more likely to be entered early because ready to achieve a commensurately high grade, rather than to ‘bank’ a Grade C which they may or may not subsequently seek to improve via retakes.

As the DfE announcement says:

‘If schools are confident that pupils will achieve well even when entered early and that early entry is therefore in the interests of the pupil, they should not need to make any changes to entry plans.’

On the other hand, we have already seen in Part One that Ofsted have included in  Subsidiary Guidance supporting inspection the advice that:

‘Inspectors should investigate whether a policy of early entry to GCSE for pupils is preventing them from making as much progress as they should, for example because…

  • The widespread use of early GCSE entry and repeated sitting of examinations has encouraged short-term gains in learning but has led to underachievement at GCSE, particularly for able pupils
  • Opportunities are missed to meet the needs of high-attaining pupils through depth of GCSE study and additional qualifications.’ (para 34)

Schools would do well to ensure that their plans to improve high attainers’ performance are reflected in any revision of their early entry policies, and vice versa.

Given the significance now attached to this issue, any measure that depends on increasing the incidence of early entry for high attainers is likely to receive close scrutiny.

Schools should not be cowed from adopting such an approach where it is clearly in the best interest of their highest attainers, but they will need strong supporting evidence that early entry will result in an A*/A grade (and ideally A*), and that appropriate progression routes are in place.

 .

16-18 Tables

The post-16 Tables are comparatively less well-developed and continue to rely exclusively on high attainment measures rather than separately delineating outcomes for high attainers.

The structure of the 2013 Tables is undergoing significant change, with separate reporting of three different performance categories, depending whether students have pursued A levels, A level and other advanced academic qualifications or advanced vocational qualifications respectively.

The 2013 entries in the table below cover only the A level strand.

 .

  2012 2013 (A levels)
High attainers measures  
Attainment None Ditto
Progress None Ditto
Destinations None Ditto (see below)
Closing the Gap None Ditto
Comparisons None Ditto
High attainment measures
  % of KS5 students (4.8%) and of A level students (7.4%) achieving 3 A levels at AAB+ in three facilitating subjects % of A level students achieving 3 A levels at AAB+ in three facilitating subjects
  % of KS5 students (7.8%) and of A level students (11.9%) achieving  3 A levels at AAB+ with two in facilitating subjects % of A level students achieving 3 A levels at AAB+ with two in facilitating subjects
  APS per student and per entry for A level only APS per A level student (FTE) and per A level entry
  APS per student and per entry for A level and other academic qualifications
  APS per student and per entry for A level and equivalent qualifications (including 4 year time series)
  A level VA (see below)

 .

New value added measures are also expected for A level and the other two performance categories, though the release of this data is said to be ‘subject to further analysis’ and no substantive detail is provided.

There is no commitment to introduce KS5 destinations data into the 2013 Tables though the Statement of Intent says:

‘We will continue to evaluate both the KS5 and employment destinations measures as part of our aim to include in future performance tables.’

‘Facilitating subjects’ continue to hold sway in the key high attainment measures, despite continuing criticism of the concept, as well as concern that a ‘three facilitating subjects’ measure is not consistent with the Russell Group’s advice.

 .

What can we learn from this comparison?

It is noticeable how little consistency there is between each set of Tables as presently formulated. High attainers feature strongly in the Secondary Tables, to a limited extent in the Primary Tables and not at all in the Post-16 Tables.

Conversely, there are several measures of high attainment in the Primary Tables and a couple in the Post-16 tables, but the Secondary Tables concentrate exclusively on generic measures. There is no measure for achievement pitched at GCSEs Grades A*/A.

Closing the gap data and comparisons data is so far entirely absent from the post-16 Tables and, perhaps less surprisingly, destinations data is absent from the Primary Tables.

Where destinations and closing the gap data is available, there is no breakdown for high attainers.

The next section will explore whether the changes proposed in the three accountability-related consultation documents are likely to bring about any greater consistency in the coverage of the respective Performance Tables and, more specifically, how they are likely to report on high attainers.

 .

Likely Impact of Accountability Reforms

At the time of writing, three staged consultations have been launched but only one has been concluded.

All three consultations are predicated on a new approach to the publication of accountability data which has three components:

  • A requirement to publish a handful of headline indicators on schools’ own websites. The secondary consultation response describes this as a ‘snapshot’ in standard format. It seems likely that this provision will be extended to primary and post-16 institutions but, as yet, this is nowhere confirmed.
  • Continued publication of Performance Tables which contain the same headline indicators, but with additional material showing how different groups of learners perform against them and how performance compares with other institutions.
  • The introduction by March 2015 of a new data portal which contains all the (performance) information held about schools (and colleges?) We are told in the secondary consultation response that:

‘It will be an easily accessible website that allows the public to search all the information we hold about schools, subject to protecting individuals’ anonymity. Respondents to the consultation argued that it would be useful to see measures showing school by school performance in vocational qualifications, the percentage of pupils achieving the top grades in GCSEs, and average grades by subject. We agree that all these measures will be of interest to many people, and the Data Portal is being designed so that parents can search for this type of information.’

There is otherwise comparatively little information about this portal in the public domain.

DfE’s Digital Strategy, published in December 2012, says that the parent School Performance Data (SDP) programme will ‘see the consolidation of 8 existing data-based services into one’ and that delivery of the programme will be staggered from the second quarter of 2014 until the final quarter of 2015. At some point within this timeline, it will absorb and replace RAISE Online.

.

.

Primary Accountability Reform

Those with a wider interest in the proposed changes to primary accountability and assessment are invited to read this analysis.  This commentary deals only with the material likely to be published through one of the three routes described above.

The primary document is comfortably the vaguest of the three consultations. However, we know that new KS2 tests – single tests for the full attainment spectrum – will be introduced in 2016, with results first appearing in the 2016 Performance Tables, likely to be published in December that year.

We are told that tests will be available in maths and in elements of English including reading (GPS is not mentioned explicitly). Writing will continue to be assessed via teacher assessment.

Published performance measures are typically described in the consultation as relating to ‘each subject’ but it seems most likely that elements of English will continue to be reported separately.

All the measures outlined below therefore apply to maths and to each tested element of English. One assumes they also apply to writing TA – and potentially to other TA outcomes too – but this is unclear.

So there may be one or two sets of measures for maths (test and potentially TA), while for English there will be somewhere between one (reading test) and five (reading and GPS tests; reading, writing and GPS TA).

For each subject/element, Performance Tables are expected to include:

  • The percentage of pupils ‘meeting the secondary readiness standard’, which will already have been introduced in the 2013 Tables. Secondary readiness will be reported using scaled scores derived from raw test marks. A score of 100 is proposed to denote the threshold of secondary readiness, so the Tables will show the percentage of a school’s learners at or above 100.
  • The corresponding average scaled scores for each school. The consultation document adopts an illustrative scale – based on the current national curriculum tests – which runs from 80 to 130. An average scaled score significantly over 100 is a positive indicator but, of itself, says nothing about the distribution of scores, or whether this outcome is attributable to prior attainment.
  • ‘How many of the school’s pupils are among the highest-attaining nationally, by…showing the percentage of pupils attaining a high scaled score’. There is no information about where these measures will be pitched in relation to the deciles that will be used to report to parents on individual learners’ performance. It might be that the cut-off for high attainers is pitched at a particular score – say 120 on the illustrative scale above – or at the 20th percentile, for example. (It seems unlikely that numbers performing in each decile will be reported in the tables though they could presumably be found via the data portal.)
  • Progress measures based on comparison of pupils’ scaled scores with the scores of other pupils with the same prior attainment. The paper gives the impression that there will be separate progress measures for each subject/element, rather than a composite measure such as that planned for secondary schools (see below). These will be derived from performance on a baseline assessment, which seems increasingly likely to be moved back from the end of KS1 to YR. We are not told what categories of prior attainment will be applied. A sophisticated gradation, based on deciles, is unlikely to be consistent with a simple baseline check for 4 year-olds. A cruder tripartite judgement of ‘primary-readiness’ is more probable, with learners categorised as ‘not yet primary ready’, ‘primary ready’ or ‘performing beyond primary readiness’.
  • Comparison of each school’s performance with other schools with similar intakes. Whether this will be undertaken in relation to all the measures above, or only selected measures, is so far unclear.
  • The attainment and progress of those in receipt of the Pupil Premium. Whether this analysis will be provided for to all the measures above also remains unclear.

All measures will be published as annual results and as three year rolling averages

The document asks respondents to comment on whether other measures should be prioritised within performance tables. It is not clear which of this data will be published on schools’ own websites as well as in performance tables.

.

Secondary accountability reform

The response to the secondary consultation confirms that schools will be required to publish five measures in a standard format on their own websites:

  • Attainment across a suite of up to eight qualifications (called ‘Attainment 8’). This suite includes: English and maths, both of which are double weighted to reflect their significance (English language will only be double weighted if combined with English literature); three further EBacc subjects (combined science double award will count as two qualifications; subjects can be drawn from the same subject area, eg three sciences or two languages); and three further qualifications, which may be arts-based, academic or approved vocational qualifications (English literature may count as one of these). The measure will be applied to learners entering fewer than eight qualifications and those entering more will have their highest graded qualifications counted in the final category. The outcome will be expressed as an average grade, but with finer gradations – exemplification in the response refers to ‘a high B grade or a low D grade’. (The consultation document noted that learners would know their APS and be able to compare it with ‘easily available local and national benchmarks’).
  • An aggregated progress measure (called ‘Progress 8’) based on the same suite of up to eight qualifications. KS4 results will be predicted on the basis of prior attainment at KS2. This involves calculating an estimated KS4 outcome based on the actual outcomes of all learners with a specified level of KS2 attainment across English and maths. (The two examples in the response are based on the existing KS2 APS methodology and averaged NC levels respectively, rather than the proposed new ‘scaled scores’ methodology outlined above. We do not know on what basis such scores would be aggregated in future for the purpose of this calculation.) Each learner’s VA score is calculated by subtracting their estimated outcome from their actual outcome. So, if a learner with given prior attainment is estimated to achieve 8C grades at GCSE and actually achieves 4Bs and 4Cs, that gives a VA score of +0.5 (4 grades over 8 subjects). The school’s average VA score is calculated by aggregating these outcomes.
  • The percentage of learners achieving a ‘pass’ grade or better in English and maths. The response uses the existing nomenclature of a C grade or better, but this will change when a new GCSE grading system is finalised. Outcomes from the Ofqual consultation – which proposed a new grading scale from 1 (low) to 8 (high) and a subsequent standards-setting consultation – should be available shortly.
  • The percentage achieving ‘good grades’ in the EBacc (the same issue applies) and
  • A destination measure, derived from the current experimental approach. This is provisional since ‘we want to be sure the statistics are robust before committing to use this…as a headline indicator’.

The response adds that the calculation process for ‘Progress 8’ is under review. It is likely to be adjusted, by calculating expected progress from the results of learners who completed KS4 three years previously, though possibly not until 2019.

There might also be a shift to calculating expected progress at subject level, which is then averaged, to reduce the likelihood of ‘schools entering learners for qualifications ‘in which it is easier to score points for the progress measure’.

A straightforward linear point scoring system is under discussion – eg 1 for a current GCSE Grade G to 8 for a current A* grade. This would give more credit to schools for higher results than the current non-linear approach, which awards a G 16 points and an A* 58 points. (this might suggest a similar adjustment to the primary APS methodology.)

Finally, the treatment of an estimated 1.2% of low attainers who enter no relevant qualifications is still under consideration. The methodology will not be finalised until Spring 2014.

Performance Tables will ‘eventually’ include the five headline indicators above (no date is provided for this).

They will also contain:

  • A value added progress measure in English and maths, showing whether learners have performed better or worse than expected given their prior attainment at KS2. (The original consultation document implied this would relate to English and maths separately and would be provided for low, middle and high attainers respectively.)
  • A comparison measure with similar schools, using the existing methodology, but further developed to give ‘an indication of disadvantage’ in each group of similar schools.
  • By implication, ‘closing the gap’ indicators showing the attainment of disadvantaged learners eligible for the Pupil Premium, the progress of those learners and ‘the in-school gap in attainment between disadvantaged pupils and their peers’. Both single year data and three year rolling averages will be published.
  • By implication there will also be further analysis of performance by high, middle and low attainers on each measure. The drafting is however very unclear, referring to the present rather than the future:

‘Performance tables give a breakdown of the performance of different pupil groups on the headline indicators. They show how well pupils with low, middle and high prior attainment perform on each measure…For each indicator, local and national benchmarks are provided to make it easier to judge each school’s performance. Using this information, parents can search for the information which is most relevant to them. For example, they can see how many pupils with high prior attainment go on to achieve the EBacc at different schools in their area’

It seems most likely that all learners judged ahead of the new ‘secondary ready’ threshold would count as high attainers, even though it would now be possible to calculate more sophisticated gradations based on deciles of performance.

Hence we would continue to have three broad categories of prior attainment: exceeding secondary ready standard, at secondary ready standard and not yet secondary ready.

However, the consultation response is silent on this matter.

We are told that material about pupils achieving the top grades at GCSE will only be available through the data portal, which means that – on this issue – the new secondary tables will continue to be out of kilter with the primary and post-16 tables.

There is one further unexplained reference in the response:

‘Confidence intervals will also be important when we present each school’s percentile ranking on the range of headline measures. For example, a school could have performed well on the Attainment 8 measure and be in the 10th percentile, with a confidence interval that indicates that the school’s true ranking is likely to lie between the 5th and 15th percentiles.’

This appears to suggest that schools will be ranked on each headline measure – but it remains unclear whether this material will be included in the Performance Tables.

There is also a beguiling reference to Ofsted:

‘In addition Ofsted may choose to specify some of these measures, for example the percentage of pupils achieving the best GCSE grades, in their inspection guidance’.

In addition:

‘Schools in which pupils make an average of one grade more progress than expected across their 8 subjects will not be inspected by Ofsted during the next academic year (unless there are exceptional circumstances, for example where there are safeguarding concerns).’

This suggests further changes to the Handbook and Subsidiary Guidance, and potentially to the Framework itself.

Changes will be introduced into the 2016 Performance Tables published in January 2017, but schools will receive information based on 2014 exam results to illustrate their performance on the new measures, and will be able to opt in to a new floor standards methodology in 2015.

 .

Post-16

This consultation proposes that performance should be reported separately at Level 2 (including continued study of English and maths for those so far without a GCSE ‘pass’ – currently graded A*-C) and Level 3.

In respect of Level 3 performance should be reported for three strands of provision: Academic (A level, AS level, IB, EPQ etc), Applied General and Technical, as has been introduced for 2013 Tables.

This analysis focuses exclusively on Level 3 Academic provision.

At Level 3, five ‘topline performance measures’ are proposed which will be included in Performance Tables (it is not stated whether institutions will also be required to publish them on their websites):

The proposed topline measures are:

  • Two principal attainment measures: average grade and average points per full-time A level student – a best 3 A levels measure is under consideration ‘to encourage substantial A level programmes’; and average grade and points score per A level entry. This is given pride of place, above the measures that rely on facilitating subjects and there is no reference to restrictions being placed on the subjects studied.
  • An aggregate KS4-5 progress measure ‘showing the progress of similar students nationally who, according to their results at the end of Key Stage 4, were of the same level of ability’ (by which they mean attainment). This sounds broadly consistent with what is proposed for the primary and secondary tables. The annex says:

‘Only students with the same prior attainment, taking the same subjects, will be compared to provide a subject score. Subject scores will then be aggregated with other academic…scores to provide an overall academic score.’

This presumably means that A level students will be compared with those with similar KS4 attainment who are taking the same A level subjects.

  • A destination measure ‘showing student progression to a positive destination’. No reference is made to the controversial question whether this will continue to distinguish Russell Group universities and Oxbridge.
  • A completion measure.

There is additionally a commitment to ‘consider how we can report the results of low, middle and high attainers similarly [to KS4] in the expanded 16-19 performance tables’ but no further clue as to how these will be devised.

A number of additional measures are also laid out, which may or may not appear in the performance tables:

  • The percentage of students achieving AAB+ grades at A level in two and in three ‘facilitating subjects’. Note that the benchmark is still AAB+, even though, from 2013-14 onwards, HEFCE’s student number control relaxation – from which this measure was originally derived – is extended from AAB+ to ABB+.
  • A ‘closing the gap’ measure which will show attainment by pupils who were eligible for Pupil Premium funding in Year 11. The annex suggests that this ‘can be compared with the top line attainment measure’ which, in the context of A level, may mean one or both of the two described above. It is not clear whether this will also be applied to the progress measure.
  • Attainment of approved level 3 maths qualifications for students who do not take A or AS level (these are under development). These will be available for teaching from 2015.

The timetable for the introduction of these reforms is not specified.

 .

Comparison of Primary, Secondary and Post-16 reforms

The table below shows the extent to which the overall proposals for performance table reform reflect a consistent application of the typology set out above.

.

Primary Secondary Post-16
Attainment % achieving secondary ready standardAverage scaled scores ‘Attainment 8’ expressed as average gradePass grade in E+MPass grades in EBacc Average grade and APS per FT A level student (potentially on best 3 A levels)Average grade and APS per A level entryAAB+ in 2 and 3 facilitating subjectsPerformance in L3 maths qualifications
Progress Averaged scaled scores compared with prior attainment ‘Progress 8’ expressed as +/- average gradeProgress in E+M Yes but no detail
Destinations None mentioned Provisionally Yes but no detail
Closing the gap Yes (indicators unclear) Yes (unclear if applied for all  indicators above Yes but no detail
Comparisons Yes (indicators unclear) Yes (with ‘indication of disadvantage’) None mentioned
Notes Reference to ‘percentile rankings on headline measures’ Unclear if facilitating subjects and L3 maths in tables

There is evidence of a shift towards greater consistency of approach compared with the current performance tables, although many of the details have yet to be clarified.

Unfortunately, this lack of clarity extends to the definition of high attainers and how their performance will be reported.

  • In the primary sector, it seems most likely that high attainers will be defined according to some yet-to-be-determined measure of ‘primary readiness’ and in the secondary sector according to the new ‘secondary-ready’ standard. One could extend the same logic into post-16 by devising a ‘sixth form ready’ standard based on performance against new-style GCSE grades. This would transpose into the new Tables the rather crude tripartite distinction in place currently at primary and secondary level, though we know the pitch will be somewhat higher. But this is little more than an educated guess. It would be quite possible to introduce a more sophisticated distinction and a narrower definition of what constitutes a high attainer, though this would be much more straightforward in the secondary sector than in primary.
  • We are equally unclear how high attainers’ performance will be reported. There is nothing explicit in the primary consultation document to explain which measures will be applied to high, middle and low attainers, Indeed, the only direct reference to such distinctions applies to Ofsted inspection:

‘Schools in which low, middle and high attaining pupils all make better than average progress will be much less likely to be inspected sooner.’

The secondary consultation response implies that all the headline measures will be applied to these three categories of prior attainment, but fails to state this explicitly, while the post-16 document doesn’t go beyond the broader commitment mentioned above.

  • As for the reporting of high attainment, the methodology for the principal  progress measures – confirmed for the primary and secondary tables and planned for post-16 – are specifically designed to discourage schools from concentrating over-much on learners on the borderline of a threshold measure. This is welcome. But, whereas, the primary tables will include new measures focused explicitly on how many of a school’s population have achieved national measures of high attainment (as expressed by high scaled scores) and high attainment measures will be retained in the new post-16 tables, the continued omission of an equivalent GCSE measure from the secondary tables seems inconsistent, even though we are told it will be possible to find this data in the accompanying portal. It is hard to understand the logic that justifies this continued inconsistency of approach, when the broader direction of travel seems very much about bring the three sets of tables more closely into line.

 .

How Should Schools Respond?

With so much change in the offing, it can be all too easy for schools to slip into a defensive, reactive mode, particularly if they are under pressure on other fronts.

There is a temptation to concentrate effort elsewhere, on the grounds that high attainers will do comparatively well with relatively little support. Most will be secure L5 performers and go on to pick up a clutch of A grades at GCSE. The opportunity cost of lifting them to L6 and converting their As into A*s may be perceived as simply too great.

On the other hand, while I can cite no hard evidence to support the contention, I have often observed that schools which are successful with their high attainers are rarely unsuccessful in other respects. It is as if support for high attainers is a litmus test of personalised education, going the extra mile and wider school effectiveness.

And there is a real opportunity for schools to get on to the front foot with this issue, given the absence of any substantive lead or guidance from the centre, or any conspicuous consensus amongst schools – or between experts, academics and service providers – over what constitutes effective practice.

Naturally schools will want to frame their response in a way that addresses their priorities and fits their particular contexts. They will need to take into account how their success is defined and reported by Ofsted and in School Performance Tables – and how this might change in the future – but they will not plan exclusively on that basis.

They must find the ‘best fit’ between the demands of the accountability regime and what is in the best interests of their learners. The regime is not intended to impose rigid conformity, and higher performing schools in particular must be allowed to trust their judgement rather than devoting themselves exclusively to these ‘one-size-fits-all’ measures of success.

The first part of this final section sets out how a school might rethink and redefine its support for high attainers from first principles – though it stops short of advocating or discussing any specific elements of effective whole school practice.

The second part draws on the analysis elsewhere in this post to inform a suggested basket of key measures from which schools might select when constructing a plan to improve the their high attainers’ performance. This is very much intended as a starting point for discussion, rather than a blueprint that all must follow.

.

Rethinking Support for High Attainers

A high attainers’ support strategy will only be completely successful if it has the full commitment of staff, learners, parents and other key stakeholders. It should extend across the whole school and all dimensions of whole school practice, including learning at home and parental and community engagement.

The best way to secure wholesale commitment is through an inclusive and transparent consultative process. The outcomes of that process are most readily captured in an accessible document that should:

  • Include a clear, comprehensive yet succinct statement of whole school policy for challenging and supporting high attainers that is meaningful and relevant to lay and professional audiences alike.
  • Incorporate a concise improvement plan that is regularly monitored and updated and that feeds into the wider school improvement plan.
  • Be published openly, so that the school’s priorities are understood and acted on by all parties – and so that prospective parents and learners can use them to inform decisions about whether to apply for admission.

In formulating a support plan, schools must consider what relationship it should have with any parallel gifted and talented education policy (or equivalent terminology adopted by the school).

It is not appropriate simply to substitute one for the other, without giving careful consideration to what might be lost in doing so.

In future, how will the school support learners with high ability but who might not realise it through high attainment? What of twice-exceptional learners, for example, and those with diverse talents that are not demonstrated through high attainment, whether in arts, sports, interpersonal skills, or any other field judged significant by the school?

Some schools may prefer to have parallel and mutually supportive policies for these two overlapping populations; others may prefer an integrated policy.

The most straightforward approach is to distinguish high attainers as a subset of the school’s wider gifted and talented population. Ignore the school of thought that suggests high attainers are somehow different, ‘bright but not gifted’.

But, if the policy is integrated, it must be clear where and how support for high attainers is distinct.

The support plan must rest on a clear definition of what constitutes a high attainer – and a potential high attainer – in the context of the school, together with explicit recognition of the gradations of high attainment within the general definition. (Ofsted’s example shows how confusion may be caused by using inconsistent terminology and a failure to define terms.)

The plan should be framed around core priorities. These will typically include:

  • A judicious blend of challenge and support for high attainers, designed to ensure that they continue to perform highly, but are not exposed to undue pressure or stress; and that their high attainment is not achieved at the expense of personal wellbeing, or wider personal development.
  • Challenge and support for learners who are not yet high attainers but might become so. (This might include the remainder of the school population, or a more narrowly defined group, depending on context and ideological preference.)
  • Targeted challenge and support for disadvantaged learners and under-represented groups. This must include those in receipt of the Pupil Premium but might also reflect gender, minority ethnic, SEN and summer-born considerations. Schools understand the complex causes of underachievement and that most underachieving learners are affected by a combination of factors. Avoid a simplistic quota-driven model. Critically, this equity-driven support must not be achieved at the expense of advantaged learners. The optimal strategy is to continue to raise standards for all high attainers, but to raise them relatively faster for those from disadvantaged backgrounds.

The plan must show how the current state will be changed into the desired future state. This necessitates:

  • A thorough review process and full statement of the baseline position, preserving a balance between the celebration of strengths and the identification of weaknesses and areas for development. Schools should not gloss over gaps in their skillset, or evidence that particular subjects/departments are weaker than others. This should not be an exercise in finding and attributing fault, but a collective endeavour undertaken in a spirit of continuous improvement.
  • A coherent set of priorities for improvement which must be SMART (Specific, Measurable, Achievable, Realistic and Timebound). A named individual should be accountable for each priority and it should be stated explicitly what staff time, budget and any other resources are allocated to securing it.
  • Improvement priorities should be aligned with a matching set of outcome measures, or success criteria, which might draw from the suggested basket set out below. These should capture the intended impact of the improvement priorities on high attainers’ performance.
  • Arrangements for regular monitoring and review. There should be capacity to manage slippage and risk and to accommodate succession planning. A senior manager should be accountable for the plan as a whole, but it should be evident how all staff and all stakeholders – including learners and parents – can contribute positively to its achievement.

If the school is sufficiently confident it should consider incorporating an explicit entitlement to challenge and support for all its high attaining learners. This should not be vague and generalised, but sharp and explicit, so that parents and learners can challenge the school and seek redress if this entitlement is not forthcoming.

.

A potential basket of key measures

What measures might schools and colleges select from when developing such plans? I have set out below some first efforts at three baskets of key measures relating to primary, secondary and post-16 respectively.

These broadly reflect the current accountability regime, though I have suggested some departures, to fill gaps or in response to known concerns.

I do not pretend that these are more than illustrative. Some of the measures beg questions about definition, a few are rather iffy and there are certainly gaps in the coverage.

But they should serve to exemplify the broad approach, as well as providing a basis for more rigorous discussion at institutional level. I don’t have all the answers and very much want to start the conversation, rather than attempting to close it off.

Planners might reasonably consider drawing one measure from each of the five areas in the typology I have set out. Failing that, they might aim for a ‘balanced scorecard’ rather than relying excessively on measures in just one or two categories.

The optimal number of measures is probably between three and five – if there are fewer than three the scope of the plan will be too narrow; if there are more than five it will be too complex.

It should be possible to develop and refine these baskets over time, to reflect ideas and suggestions from those engaging with them. I hope to revisit them in future.

They will need significant adjustment to reflect the new accountability regime, once the proposals in the three consultation documents have been implemented.

And, hopefully, the new Data Portal will make it much easier to construct the necessary measures in all five of these categories, once it is introduced from 2014.

 .

                                     PRIMARY BASKET OF INDICATORS
Attainment % of high attainers at KS1
  % of high attainers at KS2
% of high attainers achieving KS1 L3
% of high attainers achieving KS2 L5B/5A/6
Progress 100% of high attainers make expected progress in KS1
  % of high attainers making more than expected progress in KS1
  100% of high attainers make at least 2 levels of progress in KS2
  % of high attainers making more than 2/up to 3 levels of progress in KS2
Destinations % of high attainers transferring to outstanding secondary schools
  % of high attainers transferring to selective schools
Closing the gap For any of the indicators above, the FSM gap for high attainers is at least x% lower than the FSM gap for middle attainers
  For any of the indicators above, the FSM gap is closed by x%
  High attainers are representative of school population (by eg FSM, gender, ethnic background, SEN, month of birth)
Comparisons For any of the indicators above, compare with all schools located in LA area/governed by academy trust
  For any of the indicators above, compare with family of schools with broadly similar intake

 .

 .

                                            SECONDARY BASKET OF INDICATORS
Attainment % of KS2 high attainers in Y7 intake
% of KS2 high attainers achieving 5+ A*-A grades at GCSE or equivalent including GCSE English and maths
% of KS2 high attainers’ GCSE entries awarded A*/A grades, or A* grades only
% of KS2 high attainers awarded GCSE A*/A grades or A* grades via early entry
Increase in GCSE APS (Best 8) and/or improvement in average grade
Increase in GCSE APS (new Attainment 8 measure) and/or improvement in average grade
Progress 100% of KS2 high attainers making expected progress from KS2-4
  % of KS2 high attainers making more than expected progress from KS2-4
  % of KS2 high attainers making 4/5 levels of progress from KS2-4
Destinations % of KS2 high attainers transferring to outstanding sixth forms/post-16 institutions
Closing the gap For any of the indicators above, the FSM gap for high attainers is at least x% lower than the FSM gap for middle attainers
  For any of the indicators above, the FSM gap is closed by x%
  High attainers are representative of school population by eg FSM, ethnic background, SEN, month of birth
Comparisons For any of the indicators above, compare with other schools located in LA area/governed by academy trust
  For any of the indicators above, compare with a family of schools with broadly similar intake

.

 .

                                             POST-16 BASKET OF INDICATORS
Attainment % of KS2 high attainers achieving AAB+/ABB+ grades at A level (whether or not in facilitating subjects)
% of KS2 high attainers’ A level entries awarded B/A/A* or A*/A or A* grades only
Increase in A level APS (best 3) and/or improvement in average grade
Progress % with GCSE A* achieving A level grade A* in same subject
  % with 5+ GCSEs at A*/A including English and maths achieving 3+ A levels at AAB/ABB
Destinations % transferring to high tariff undergraduate degree courses
  % transferring to selective/Russell Group/Oxbridge universities
Closing the gap For any of the indicators above, the FSM gap for high attainers is % lower than the FSM gap for middle attainers
  For any of the indicators above, the FSM gap is closed by x%
  High attainers are representative of sixth form population by eg FSM, gender, ethnic background, SEN, month of birth
Comparisons For any of the indicators above, compare with other schools located in LA area/governed academy trust
  For any of the indicators above, compare with a family of schools/colleges with broadly similar intake

.

Conclusion

In this second part of the post, I have adopted a simple typology of performance measures to explain:

  • Imminent changes in how the Performance Tables address high attainers and high attainment, highlighting key differences between sectors and outlining the direction of travel towards longer term reform.
  • What is known and what is still unknown about the focus on high attainers in new performance table arrangements, mostly scheduled for implementation from 2016, and to what extent these reforms will introduce a more robust approach and greater consistency between sectors.
  • How, pending those longer term reforms, schools and colleges might set about developing a high attainers’ support plan, so seizing the initiative and skirting around some of the difficulties presented by revisions to the inspection guidance.
  • How such support plans might be constructed around a basket of key outcome measures, so that they are a. focused explicitly on improvements in institutional performance as well as improved provision for high attaining learners and b. broadly reflect performance table measures without requiring slavish adherence.

This post is very much a work in progress, striving as it does to pin down a moving target while also setting out a basic support framework with universal application. I am unhappy with some aspects of this first edition and will aim to eliminate its shortcomings in a future iteration. All suggestions for improvement are welcome.

.

GP

October 2013