Maths Mastery: Evidence versus Spin

.

On Friday 13 February, the Education Endowment Foundation (EEF) published the long-awaited evaluation reports of two randomised control trials (RCTs) of Mathematics Mastery, an Ark-sponsored programme and recipient of one of the EEF’s first tranche of awards back in 2011.

Inside-out_torus_(animated,_small)Inside-out_torus_(animated,_small)EEF, Ark and Mathematics Mastery each published a press release to mark the occasion but, given the timing, none of these attracted attention from journalists and were discussed only briefly on social media.

The main purpose of this post is to distinguish evidence from spin, to establish exactly what the evaluations tell us – and what provisos should be attached to those findings.

The post is organised into three main sections which deal respectively with:

  • Background to Mathematics Mastery
  • What the evaluation reports tell us and
  • What the press releases claim

The conclusion sets out my best effort at a balanced summary of the main findings. (There is a page jump here for those who prefer to cut to the chase.)

This post is written by a non-statistician for a lay audience. I look to specialist readers to set me straight if I have misinterpreted any statistical techniques or findings,

What was published?

On Friday 13 February the EEF published six different documents relevant to the evaluation:

  • A press release: ‘Low-cost internet-based programme found to considerably improve reading ability of year 7 pupils’.
  • A blog post: ‘Today’s findings: impact, no impact and inconclusive – a normal distribution of findings’.
  • An updated Maths Mastery home page (also published as a pdf Project Summary in a slightly different format).

The last three of these were written by the Independent Evaluators – Jerrim and Vignoles (et al) – employed through the UCL Institute of Education.

The Evaluators also refer to ‘a working paper documenting results from both trials’ available in early 2015 from http://ideas.repec.org/s/qss/dqsswp.html and www.johnjerrim.com. At the time of writing this is not yet available.

Press releases were issued on the same day by:

All of the materials published to date are included in the analysis below.

Background to Maths Mastery

What is Maths Mastery?

According to the NCETM (October 2014) the mastery approach in mathematics is characterised by certain common principles:

‘Teachers reinforce an expectation that all pupils are capable of achieving high standards in mathematics.

  • The large majority of pupils progress through the curriculum content at the same pace. Differentiation is achieved by emphasising deep knowledge and through individual support and intervention.
  • Teaching is underpinned by methodical curriculum design and supported by carefully crafted lessons and resources to foster deep conceptual and procedural knowledge.
  • Practice and consolidation play a central role. Carefully designed variation within this builds fluency and understanding of underlying mathematical concepts in tandem.
  • Teachers use precise questioning in class to test conceptual and procedural knowledge, and assess pupils regularly to identify those requiring intervention so that all pupils keep up.

The intention of these approaches is to provide all children with full access to the curriculum, enabling them to achieve confidence and competence – ‘mastery’ – in mathematics, rather than many failing to develop the maths skills they need for the future.’

The NCETM paper itemises six key features, which I paraphrase as:

  • Curriculum design: Relatively small, sequenced steps which must each be mastered before learners move to the next stage. Fundamental skills and knowledge are secured first and these often need extensive attention.
  • Teaching resources: A ‘coherent programme of high-quality teaching materials’ supports classroom teaching. There is particular emphasis on ‘developing deep structural knowledge and the ability to make connections’. The materials may include ‘high-quality textbooks’.
  • Lesson design: Often involves input from colleagues drawing on classroom observation. Plans set out in detail ‘well-tested methods’ of teaching the topic. They include teacher explanations and questions for learners.
  • Teaching methods: Learners work on the same tasks. Concepts are often explored together. Technical proficiency and conceptual understanding are developed in parallel.
  • Pupil support and differentiation: Is provided through support and intervention rather than through the topics taught, particularly at early stages. High attainers are ‘challenged through more demanding problems which deepen their knowledge of the same content’. Issues are addressed through ‘rapid intervention’ commonly undertaken the same day.
  • Productivity and practice: Fluency is developed from deep knowledge and ‘intelligent practice’. Early learning of multiplication tables is expected. The capacity to recall facts from long term memory is also important.

Its Director published a blog post (October 2014) arguing that our present approach to differentiation has ‘a very negative effect’ on mathematical attainment and that this is ‘one of the root causes’ of our performance in PISA and TIMSS.

This is because it negatively affects the ‘mindset’ of low attainers and high attainers alike. Additionally, low attainers are insufficiently challenged and get further behind because ‘they are missing out on some of the curriculum’. Meanwhile high attainers are racing ahead without developing fluency and deep understanding.

He claims that these problems can be avoided through a mastery approach:

‘Instead, countries employing a mastery approach expose almost all of the children to the same curriculum content at the same pace, allowing them all full access to the curriculum by focusing on developing deep understanding and secure fluency with facts and procedures, and providing differentiation by offering rapid support and intervention to address each individual pupil’s needs.’

But unfortunately he stops short of explaining how, for high attainers, exclusive focus on depth is preferable to a richer blend of breadth, depth and pace, combined according to each learner’s needs.

NCETM is careful not to suggest that mastery is primarily focused on improving the performance of low-attaining learners.

It has published separate guidance on High Attaining Pupils in Primary Schools (registration required), which advocates a more balanced approach, although that predates this newfound commitment to mastery.

NCETM is funded by the Department for Education. Some of the comments on the Director’s blog post complain that it is losing credibility by operating as a cheerleader for Government policy.

Ark’s involvement

Ark is an education charity and multi-academy trust with an enviable reputation.

It builds its approach on six key principles, one of which is ‘Depth before breadth’:

‘When pupils secure firm foundations in English and mathematics, they find the rest of the curriculum far easier to access. That’s why we prioritise depth in these subjects, giving pupils the best chance of academic success. To support fully our pupils’ achievement in maths, we have developed the TES Award winning Mathematics Mastery programme, a highly-effective curriculum and teaching approach inspired by pupil success in Singapore and endorsed by Ofsted. We teach Mathematics Mastery in all our primary schools and at Key Stage 3 in a selection of our secondary schools. It is also being implemented in over 170 schools beyond our network.’

Ark’s 2014 Annual Report identifies five priorities for 2014/15, one of which is:

‘…developing curricula to help ensure our pupils are well prepared as they go through school… codifying our approach to early years and, building on the success of Maths Mastery, piloting an English Mastery programme…’

Mathematics Mastery is a charity in its own right. Its website lists 15 staff, a high-powered advisory group and three partner organisations:  Ark, the EEF (presumably by virtue of the funded evaluation) and the ‘Department for Education and the Mayor of London’ (presumably by virtue of support from the London Schools Excellence Fund).

NCETM’s Director sits on Mathematics Mastery’s Advisory Board.

Ark’s Chief Executive is a member of the EEF’s Advisory Board.

Development of Ark’s Maths Mastery programme

According to this 2012 report from Reform, which features Maths Mastery as a case study, it originated in 2010:

‘The development of Mathematics Mastery stemmed from collaboration between six ARK primary academies in Greater London, and the mathematics departments in seven separate ARK secondary academies in Greater London, Portsmouth and Birmingham. Representatives from ARK visited Singapore to explore the country’s approach first-hand, and Dr Yeap Ban Har, Singapore’s leading expert in maths teaching, visited King Solomon Academy in June 2011.’

In October 2011, EEF awarded Ark a grant of £600,000 for Maths Mastery, one of its first four awards.

The EEF’s press release says:

‘The third grant will support an innovative and highly effective approach to teaching children maths called Mathematics Mastery, which originated in Singapore. The programme – run by ARK Schools, the Academies sponsor, which is also supporting the project – will receive £600,000 over the next four years to reach at least 50 disadvantaged primary and secondary schools.’

Ark’s press release adds:

‘ARK Schools has been awarded a major grant by the Education Endowment Foundation (EEF) to further develop and roll out its Mathematics Mastery programme, an innovative and highly effective approach to teaching children maths based on Singapore maths teaching. The £600,000 grant will enable ARK to launch the programme and related professional development training to improve maths teaching in at least 50 disadvantaged primary and secondary schools.

The funding will enable ARK Schools to write a UK mathematics mastery programme based on the experience of teaching the pilot programme in ARK’s academies. ARK intends to complete the development of its primary modules for use from Sept 2012 and its secondary modules for use from September 2013. In parallel ARK is developing professional training and implementation support for schools outside the ARK network.’

The project home page on EEF’s site now says the total project cost is £774,000. It may be that the balance of £174,000 is the fee paid to the independent evaluators.

This 2012 information sheet says all Ark primary schools would adopt Maths Mastery from September 2012, and that its secondary schools have also devised a KS3 programme.

It describes the launch of a Primary Pioneer Programme from September 2012 and a Secondary Pioneer Programme from September 2013. These will form the cohorts to be evaluated by the EEF.

In 2013, Ark was awarded a grant of £617,375 from the Mayor of London’s London Schools Excellence Fund for the London Primary Schools Mathematics Mastery Project.

This is to support the introduction of Mastery in 120 primary schools spread across 18 London boroughs. (Another source gives the grant as £595,000)

It will be interesting to see whether Maths Mastery (or English Mastery) features in the Excellence Fund’s latest project to increase primary attainment in literacy and numeracy. The outcomes of the EEF evaluations may be relevant to that impending decision.

Ark’s Mathematics Mastery today

The Mathematics Mastery website advertises a branded variant of the mastery model, derived from a tripartite ‘holistic vision’:

  • Deep understanding, through a curriculum that combines universal high expectations with spending more time on fewer topics and heavy emphasis on problem-solving.
  • Integrated professional development through workshops, visits, coaching and mentoring and ‘access to exclusive online teaching and learning materials, including lesson guides for each week’.
  • Teacher collaboration – primary schools are allocated a geographical cluster of 4-6 schools while secondary schools attend a ‘national collaboration event’. There is also an online dimension.

It offers primary and secondary programmes.

The primary programme has three particular features: use of objects and pictures prior to the introduction of symbols; a structured approach to the development of mathematical vocabulary; and heavy emphasis on problem-solving.

It involves one-day training sessions for school leaders, for the Maths Mastery lead and those new to teaching it, and for teachers undertaking the programme in each year group. Each school receives two support visits and attends three local cluster meetings.

Problem-solving is also one of three listed features of the secondary programme. The other two are fewer topics undertaken in greater depth, plus joint lesson planning and departmental workshops.

There are two full training days, one for the Maths Mastery lead and one for the maths department plus an evening session for senior leadership. Each school receives two support visits and attends three national collaborative meetings. They must hold an hour-long departmental workshop each week and commit to sharing resources online.

Both primary and secondary schools are encouraged to launch the programme across Year 1/7 and then roll it upwards ‘over several years’.

The website is not entirely clear but it appears that Maths Mastery itself is being rolled out a year at a time, so even the original primary early adopters will have provision only up to Year 3 and are scheduled to introduce provision for Year 4 next academic year. In the secondary sector, activity currently seems confined to KS3, and predominantly to Year 7.

The number of participating schools is increasing steadily but is still very small.

The most recent figures I could find are 192 (Maths Mastery, November 2014) or 193 – 142 primary and 51 secondary (Ark 2015).

One assumes that this total includes

  • An original tranche of 30 primary ‘early adopters’ including 21 not managed by Ark
  • 60 or so primary and secondary ‘Pioneer Schools’ within the EEF evaluations (ie the schools undertaking the intervention but not those forming the control group, unless they have subsequently opted to take up the programme)
  • The 120 primary schools in the London project
  • Primary and secondary schools recruited outwith the London and EEF projects, either alongside them or subsequently.

But the organisation does not provide a detailed breakdown, or show how these different subsets overlap.

They are particularly coy about the cost. There is nothing about this on the website.

The EEF evaluation reports say that 2FE primary schools and secondary schools will pay ‘an upfront cost of £6,000 for participating in the programme’.

With the addition of staff time for training, the per pupil cost for the initial year is estimated as £127 for primary schools and £50 for secondary schools.

The primary report adds:

‘In subsequent years schools are able to opt for different pathways depending on the amount of support and training they wish to choose; they also have ongoing access to the curriculum materials for additional year groups. The per pupil cost therefore reduces considerably, to below £30 per pupil for additional year groups.’

In EEF terms this is deemed a low cost intervention, although an outlay of such magnitude is a significant burden for primary schools, particularly when funding is under pressure, and might be expected to act as a brake on participation.

Further coyness is evident in respect of statutory assessment outcomes. Some details are provided for individual schools, but there is precious little about the whole cohort.

All I could find was this table in the Primary Yearbook 2014-15.

.

EEF maths mastery performance

It suggests somewhat better achievement at KS1 L2b and L3c than the national average but, there is no information about other Levels and, of course, the sample is not representative, so the comparison is of limited value.

An absence of more sophisticated analysis – combined with the impression of limited transparency for those not yet inside the programme – is likely to act as a second brake on participation.

There is a reference to high attainers in the FAQ on the website:

‘The Mathematics Mastery curriculum emphasises stretching through depth of understanding rather than giving the top end of pupils [sic] new procedures to cover.

Problem solving is central to Mathematics Mastery. The great thing about the problems is that students can take them as far as they can, so those children who grasp the basics quickly can explore tasks further. There is also differentiation in the methods used, with top-end pupils typically moving to abstract numbers more quickly and spending less time with concrete manipulatives or bar models. There are extension ideas and support notes provided with the tasks to help you with this.

A range of schools are currently piloting the programme, which is working well in mixed-ability classes, as well as in schools that have set groups.’

The same unanswered questions arise as with the NCETM statement above. Is ‘Maths Mastery’ primarily focused on the ‘long tail’, potentially at the expense of high attainers?

The IoE evaluators think so. The primary evaluation report says that:

‘Mathematics Mastery intervention is particularly concerned with the ‘mastery’ of basic skills, and raising the attainment of low achievers.’

It would be helpful to have clarity on this point.

.

How influential is Maths Mastery?

Extremely influential.

Much educational and political capital has already been invested in Maths Mastery, hence the peculiar significance of the results contained in the evaluation reports.

The National Curriculum Expert Panel espoused mastery in its ‘Framework for the National Curriculum‘ (December 2011), while ducking the consequences for ‘stretch and challenge’ for high attainers – so creating a tension that remains unresolved to this day.

Meanwhile, the mastery approach has already influenced the new maths programme of study, as the NCETM document makes clear:

‘The 2014 national curriculum for mathematics has been designed to raise standards in maths, with the aim that the large majority of pupils will achieve mastery of the subject…

… For many schools and teachers the shift to this ‘mastery curriculum’ will be a significant one. It will require new approaches to lesson design, teaching, use of resources and support for pupils.’

Maths Mastery confirms that its Director was on the drafting team.

Mastery is also embedded in the national collaborative projects being undertaken through the Maths Hubs. Maths Mastery is one of four national partners in the Hubs initiative.

Ministers have endorsed the Ark programme in their speeches. In April 2014, Truss said:

‘The mastery model of learning places the emphasis on understanding core concepts. It’s associated with countries like Singapore, who have very high-performing pupils.

And in this country, Ark, the academy chain, took it on and developed it.

Ark run training days for maths departments and heads of maths from other schools.

They organise support visits, and share plans and ideas online with other teachers, and share their learning with a cluster of other schools.

It’s a very practical model. We know not every school will have the time or inclination to develop its very own programmes – a small rural school, say, or single-class primary schools.

But in maths mastery, a big chain like Ark took the lead, and made it straightforward for other schools to adopt their model. They maintain an online community – which is a cheap, quick way of keeping up with the best teaching approaches.

That’s the sort of innovation that’s possible.

Of course the important thing is the results. The programme is being evaluated so that when the results come out headteachers will be able to look at it and see if it represents good value.’

In June 2014 she said:

‘This idea of mastery is starting to take hold in classrooms in England. Led by evidence of what works, teachers and schools have sought out these programmes and techniques that have been pioneered in China and East Asia….

…With the Ark Schools Maths Mastery programme, more than 100 primary and secondary schools have joined forces to transform their pupils’ experiences of maths – and more are joining all the time. It’s a whole school programme focused on setting high expectations for all pupils – not believing that some just can’t do it. The programme has already achieved excellent results in other countries.’

Several reputations are being built upon Maths Mastery, many jobs depend upon it and large sums have been invested.

It has the explicit support of one of the country’s foremost academy chains and is already impacting on national curriculum and assessment policy (including the recent consultation on performance indicators for statutory teacher assessment).

Negative or neutral evaluations could have significant consequences for all the key players and are unlikely to encourage new schools to join the Programme.

Hence there is pressure in the system for positive outcomes – hence the significance of spin.

What the EEF evaluations tell us

.

Evaluation Protocols

EEF published separate Protocols for the primary and secondary evaluations in April 2013. These are broadly in line with the approach set out in the final evaluation reports, except that both refer much more explicitly to subsequent longitudinal evaluation:

‘In May/June 2017/18 children in treatment and control schools will sit key stage 2 maths exams. The IoE team will examine the long–run effectiveness of the Maths Mastery programme by investigating differences in school average maths test scores between treatment and control group. This information will be taken from the National Pupil Database, which the EEF will link to children’s Maths Mastery test scores (collected in 2012 and 2013)’.

‘In May/June 2018 children in treatment and control schools will sit national maths exams. The IoE team will examine the long – run effectiveness of the Maths Mastery programme by investigating differences in average maths test scores between treatment and control group. This information will be taken from the National Pupil Database, which the EEF will link to children’s Maths Mastery test scores (collected in 2013 and 2014) by NATCEN.’

It is not clear whether the intention is to preserve the integrity of the intervention and control groups until the former have rolled out Mastery to all year groups, or simply to evaluate the long-term effects of the initial one-year interventions, allowing intervention schools to drop Mastery and control schools to adopt it, entirely as they wish.

EEF Maths Mastery Project Homepage

The EEF’s updated Maths Mastery homepage has been revised to reflect the outcomes of the evaluations. It provides the most accessible summary of those outcomes.

It offers four key conclusions (my emphases):

  • ‘On average, pupils in schools adopting Mathematics Mastery made a small amount more progress than pupils in schools that did not. The effect detected was statistically significant, which means that it is likely that that improvement was caused by the programme.’
  • ‘It is unclear whether the programme had a different impact on pupils eligible for free school meals, or on pupils with higher or lower attainment.’
  • ‘Given the low per-pupil cost, Mathematics Mastery may represent a cost-effective change for schools to consider.’
  • ‘The evaluations assessed the impact of the programme in its first year of adoption. It would be worthwhile to track the medium and long-term impact of the approach.’

A table is supplied showing the effect sizes and confidence intervals for overall impact (primary and secondary together), and for the primary and secondary interventions separately.

EEF table 1 Capture

.

The support materials for the EEF’s toolkit help to explain these judgements.

About the Toolkit tells us that:

‘Average impact is estimated in terms of the additional months’ progress you might expect pupils to make as a result of an approach being used in school, taking average pupil progress over a year as a benchmark.

For example, research summarised in the Toolkit shows that improving the quality of feedback provided to pupils has an average impact of eight months. This means that pupils in a class where high quality feedback is provided will make on average eight months more progress over the course of a year compared to another class of pupils who were performing at the same level at the start of the year. At the end of the year the average pupil in a class of 25 pupils in the feedback group would now be equivalent to the 6th best pupil in the control class having made 20 months progress over the year, compared to an average of 12 months in the other class.’

There is another table showing us how to interpret this scale

EEF table 2 Capture

.

We can see from this that:

  • The overall Maths Mastery impact of +0.073 is towards the upper end of the ‘1 months progress’ category.
  • The ‘primary vs comparison’ impact of +0.10 just scrapes into the ‘2 months progress’ category.
  • The secondary vs comparison impact of +0.06 is towards the middle of the ‘1 months progress category’

All three are officially classed as ‘Low Effect’.

If we compare the effect size attributable to Maths Mastery with others in the Toolkit, it is evident that it ranks slightly above school uniform and slightly below learning styles.

A subsequent section explains that the overall impact rating is dependent on meta-analysis (again my emphases):

‘The findings from the individual trials have been combined using an approach called “meta-analysis”. Meta-analysis can lead to a more accurate estimate of an intervention’s effect. However, it is also important to note that care is needed in interpreting meta-analysed findings.’

But we are not told how, in light of this, we are to exercise care in interpreting this particular finding. There are no explicit ‘health warnings’ attached to it.

The homepage does tell us that:

‘Due to the ages of pupils who participated in the individual trials, the headline findings noted here are more likely to be predictive of programme’s impact on pupils in primary school than on pupils in secondary school.’

It also offers an explanation of why the effects generated from these trials are so small compared with those for earlier studies:

‘The findings were substantially lower than the average effects seen in the existing literature on of “mastery approaches”. A possible explanation for this is that many previous studies were conducted in the United States in the 1970s and 80s, so may overstate the possible impact in English schools today. An alternative explanation is that the Mathematics Mastery programme differed from some examples of mastery learning previously studied. For example classes following the Mathematics Mastery approach did not delay starting new topics until a high level of proficiency had been achieved by all students, which was a key feature in a number of many apparently effective programmes.’

 

There is clearly an issue with the 95% confidence intervals supplied in the first table above. 

The Technical Appendices to the Toolkit say:

‘For those concerned with statistical significance, it is still readily apparent in the confidence intervals surrounding an effect size. If the confidence interval includes zero, then the effect size would be considered not to have reached conventional statistical significance.’ (p6)

The table indicates that the lower confidence interval is zero or lower in all three cases, meaning that none of these findings may be statistically significant.

However, the homepage claims that the overall impact of both interventions, when combined through meta-analysis, is statistically significant.

And it fails entirely to mention that the impact of the both the primary and the secondary interventions separately are statistically insignificant.

The explanation of the attribution of statistical significance to the two evaluations combined is that, whereas the homepage gives confidence intervals to two decimal places, the reports calculate them to a third decimal place.

This gives a lower value of 0.004 (ie four thousandths above zero).

This can be seen from the table annexed to the primary and secondary reports and included in the ‘Overarching Summary Report’

EEF maths mastery 3 decimal places Capture

.

The distinction is marginal, to say the least. Indeed, the Evaluation Reports say:

‘…the pooled effect size of 0.073 is just significantly different from zero at conventional thresholds’

Moreover, notice that the introduction of a third decimal place drags the primary effect size down to 0.099, officially consigning it to the ‘one month’s progress’ category rather than the two months quoted above.

This might appear to be dancing on the head of a statistical pin but, as we shall see later, the spin value of statistical significance is huge!

Overall there is a lack of clarity here that cannot be attributed entirely to the necessity for brevity. The attempt to conflate subtly different outcomes from the separate primary and secondary evaluations has masked these distinctions and distorted the overall assessment.

.

The full reports add some further interesting details which are summarised in the sections below.

Primary Evaluation Report 

EEF maths mastery table 4

Key points:

  • In both the primary and secondary reports, additional reasons are given for why the effects from these evaluations are so much smaller than those from previous studies. These include the fact that:

‘…some studies included in the mastery section of the toolkit show small or no effects, suggesting that making mastery learning work effectively in all circumstances is challenging.’

The overall conclusion is an indirect criticism of the Toolkit, noting as it does that ‘the relevance of such evidence for contemporary education policy in England…may be limited’.

  • The RCT was undertaken across two academic years: In AY2012/13, 40 schools (Cohort A) were involved. Of these, 20 were randomly allocated the intervention and 20 the control. In AY2013/14, 50 schools (Cohort B) participated, 25 allocated the intervention and 25 the control. After the trial, control schools in Cohort A were free to pursue Maths Mastery. (The report does not mention whether this also applied to Cohort B.) It is not clear how subsequent longitudinal evaluation will be affected by such leakage from the control group.
  • The schools participating in the trial schools were recruited by Ark. They had to be state-funded and not already undertaking Maths Mastery:

‘Schools were therefore purposefully selected—they cannot be considered a randomly chosen sample from a well-defined population. The majority of schools participating in the trial were from London or the South East.’

  • Unlike the secondary evaluation, no process evaluation was conducted so it is not possible to determine the extent to which schools adhered to the prescribed programme. 
  • Baseline tests were administered after allocation between intervention and control, at the beginning of each academic year. Pupils were tested again in July. Evaluators used the Number Knowledge Test (NKT) for this purpose. The report discusses reasons why this might not be an accurate predictor of subsequent maths attainment and whether it is so closely related to the intervention as to be ‘a questionable measure of the success of the trial’. The discussion suggests that there were potential advantages to both the intervention and control groups but does not say whether one outweighed the other. 
  • The results of the post-test are summarised thus:

‘Children who received the Mathematics Mastery intervention scored, on average, +0.10 standard deviations higher on the post-test. This, however, only reached statistical significance at the 10% level (t = 1.82; p = 0.07), with the 95% confidence interval ranging from -0.01 to +0.21. Within Cohort A, children in the treatment group scored (on average) +0.09 standard deviations above those children in the control group (confidence interval -0.06 to +0.24). The analogous effect in Cohort B was +0.10 (confidence interval -0.05 to 0.26). Consequently, although the Mathematics Mastery intervention may have had a small positive effect on children’s test scores, it is not possible to rule out sampling variation as an explanation.’

  • The comparison of pre-test and post-test results provides any evidence of differential effects for those with lower or higher prior attainment:

‘Estimates are again presented in terms of effect sizes. The interaction effect is not significantly different from zero, with the 95% confidence interval ranging from -0.01 to +0.02. Thus there is little evidence that the effect of Mathematics Mastery differs between children with different levels of prior achievement.’

The Report adds:

‘Recall that the Mathematics Mastery intervention is particularly concerned with the ‘mastery’ of basic skills, and raising the attainment of low achievers. Thus one might anticipate the intervention to be particularly effective in the bottom half of the test score distribution. There is some, but relatively little, evidence that the intervention was less effective for the bottom half of the test distribution.

So, on this evidence, Maths Mastery is no more effective for the low achievers it is intended to help most. This is somewhat different to the suggestion on the homepage that the answer given to this question is ‘unclear’.

Several limitations are discussed, but it is important to note that they are phrased in hypothetical terms:

  • Pupils’ progress was evaluated after one academic year::

’This may be considered a relatively small ‘dose’ of the Mathematics Mastery programme’.

  • The intervention introduced a new approach to schools, so there was a learning curve which control schools did not experience:

‘With more experience teaching the programme it is possible that teachers would become more effective in implementing it.’

  • The test may favour either control schools or intervention schools.
  • Participating schools volunteered to take part, so it is not possible to say whether similar effects would be found in all schools.
  • It was not possible to control for balance – eg by ethnic background and FSM eligibility – between intervention and control. [This is now feasible so could potentially be undertaken retrospectively to check there was no imbalance.]

Under ‘Interpretation’, the report says:

‘Within the context of the wider educational literature, the effect size reported (0.10 standard deviations) would typically be considered ‘small’….

Yet, despite the modest and statistically insignificant effect, the Mathematics Mastery intervention has shown some promise.’

The phrase ‘some promise’ is justified by reference to the meta-analysis, the cost effectiveness (a small effect size for a low cost is preferable to the same outcome for a higher cost) and the fact that the impact of the entire programme has not yet been evaluated

‘Third, children are likely to follow the Mathematics Mastery programme for a number of years (perhaps throughout primary school), whereas this evaluation has considered the impact of just the first year of the programme. Long-run effects after sustained exposure to the programme could be significantly higher, and will be assessed in a follow-up study using Key Stage 2 data.’

This is the only reference to a follow-up study. It is less definite than the statement in the assessment protocol and there is no further explanation of how this will be managed, especially given potential ‘leakage’ from the control group.

Secondary Evaluation Report

EEF maths mastery table 5

Key points:

  • 50 schools were recruited to participate in the RCT during AY2013/14, with 25 randomly allocated to intervention and control. All Year 7 pupils within the former experienced the intervention.  As in the primary trial, control schools were eligible to access the programme after the end of the trial year. Interestingly, 3 of the 25 intervention schools (12%) dropped out before the end of the year – their reasons are not recorded. 
  • As in the primary trial, Ark recruited the participating schools – which had to be state-funded and new to Maths Mastery. Since schools were deliberately selected they could not be considered a random sample. The report notes:

‘Trial participants, on average, performed less well in their KS1 and KS2 examinations than the state school population as a whole. For instance, their KS1 average points scores (and KS2 maths test scores) were approximately 0.2 standard deviations (0.1 standard deviations) below the population mean. This seems to be driven, at least in part, by the fact that the trial particularly under-represented high achievers (relative to the population). For instance, just 12% of children participating in the trial were awarded Level 3 in their Key Stage 1 maths test, compared to 19% of all state school pupils in England.’

  • KS1 and KS2 tests were used to baseline. The Progress in Maths (PiM) test was used to assess pupils at the end of the year. But about 40% of the questions cover content not included in the Y7 maths mastery curriculum, which disadvantaged them relative to the control group. PiM also includes a calculator section although calculators are not used in Year 7 of Maths Mastery. It was agreed that breakdowns of results would be supplied to account for this.
  • On the basis of overall test results:

‘Children who received the Mathematics Mastery intervention scored, on average, +0.055 standard deviations higher on the PiM post-test. This did not reach statistical significance at conventional thresholds (t = 1.20; p = 0.24), with the 95% confidence interval ranging from –0.037 to +0.147. Turning to the FSM-only sample, the estimated effect size is +0.066 with the 95% confidence interval ranging from –0.037 to +0.169 (p = 0.21). Moreover, we also estimated a model including a FSM-by intervention interaction. Results suggested there was little evidence of heterogeneous intervention effects by FSM. Consequently, although the Mathematics Mastery intervention may have had a small positive effect on overall PiM test scores, one cannot rule out the possibility that this finding is due to sampling variation.

  • When the breakdowns were analysed:

‘As perhaps expected, the Mathematics Mastery intervention did not have any impact upon children’s performance on questions covering topics outside the Mathematics Mastery curriculum. Indeed, the estimated intervention effect is essentially zero (effect size = –0.003). In contrast, the intervention had a more pronounced effect upon material that was focused upon within the Mathematics Mastery curriculum (effect size = 0.100), just reaching statistical significance at the 5% level (t = 2.15; p = 0.04)

  • The only analysis of the comparative performance of high and low attainers is tied to the parts of the test not requiring use of a calculator. It suggests a noticeably smaller effect in the top half of the attainment distribution, with no statistical significance above the 55th This is substantively different to the finding in the primary evaluation, and it begs the question whether secondary Maths Mastery needs adjustment to make it more suitable for high attainers.
  • A process evaluation was focused principally on 5 schools from the intervention group. Focus group discussions were held before the intervention and again towards the end. Telephone interviews were conducted and lessons observed. The sample was selected to ensure different sizes of school, FSM intake and schools achieving both poor and good progress in maths according to their most recent inspection report. One of the recommendations is that:

The intervention should consider how it might give more advice and support with respect to differentiation.’

  • The process evaluation adds further detail about suitability for high attainers:

‘Another school [E] also commented that the materials were also not sufficiently challenging for the highest-attaining children, who were frustrated by revisiting at length the same topics they had already encountered at primary school. Although this observation was also made in other schools, it was generally felt that the children gradually began to realise that they were in fact enjoying the subject more by gaining extra understanding.’

It is not clear whether this latter comment also extends to the high attainers!

A similar set of limitations is explored in similar language to that used in the primary report.

Under ‘Interpretation’ the report says:

‘Although point estimates were consistent with a small, positive gain, the study did not have sufficient statistical power to rule out chance as an explanation. Within the context of the wider educational literature, the effect size reported (less than 0.10 standard deviations) would typically be considered ‘small’…

But, as in the primary report, it detects ‘some promise’ on the same grounds. There is a similar speculative reference to longitudinal evaluation.

.

Press releases and blogs

. 

EEF press release

There is a certain irony in the fact that ‘unlucky’ Friday 13 February was the day selected by the EEF to release these rather disappointing reports.

But Friday is typically the day selected by communications people to release educational news that is most likely to generate negative media coverage – and a Friday immediately before a school holiday is a particularly favoured time to do so, presumably because fewer journalists and social media users are active.

Unfortunately, the practice is at risk of becoming self-defeating, since everyone now expects bad news on a Friday, whereas they might be rather less alert on a busier day earlier in the week.

On this occasion Thursday was an exceptionally busy day for education news, with reaction to Miliband’s speech and a raft of Coalition announcements designed to divert attention from it. With the benefit of hindsight, Thursday might have been a better choice.

The EEF’s press release dealt with evaluation reports on nine separate projects, so increasing the probability that attention would be diverted away from Maths Mastery.

It led on a different evaluation report which generated more positive findings – the EEF seems increasingly sensitive to concerns that too many of the RCTs it sponsors are showing negligible or no positive effect, presumably because the value-for-money police may be inclined to turn their beady eye upon the Foundation itself.

But perhaps it also did so because Maths Mastery’s relatively poor performance was otherwise the story most likely to attract the attention of more informed journalists and commentators.

On the other hand, Maths Mastery was given second billing:

‘Also published today are the results of Mathematics Mastery, a whole-school approach which aims to deepen pupils’ conceptual understanding of key mathematical ideas. Compared to traditional curricula, fewer topics are covered in more depth and greater emphasis is placed on problem solving and encouraging mathematical thinking. The EEF trials found that pupils following the Mathematics Mastery programme made an additional month’s progress over a period of a year.’

.

.

EEF blog post

Later on 13 February EEF released a blog post written by a senior analyst which mentions Maths Mastery in the following terms:

Another finding of note is the small positive impact of teaching children fewer mathematical concepts, but covering them in greater depth to ensure ‘mastery’. The EEF’s evaluation of Mathematics Mastery will make fascinating reading for headteachers contemplating introducing this approach into their school. Of course, the true value of this method may only be evident in years to come as children are able to draw on their secure mathematical foundations to tackle more complex problems.’

EEF is consistently reporting a small positive impact but, as we have seen, this is rather economical with the truth. It deserves some qualification.

More interestingly though, the post adds (my emphases):

‘Our commitment as an organisation is not only to build the strength of the evidence base in education, across key stages, topics, approaches and techniques, but also ensure that the key messages emerging from the research are synthesised and communicated clearly to teachers and school leaders so that evidence can form a central pillar of how decisions are made in schools.

We have already begun this work, driven by the messages from our published trials as well as the existing evidence base. How teaching assistants can be used to best effect, important lessons in literacy at the transition from primary to secondary, and which principles should underpin approaches on encouraging children in reading for pleasure are all issues that have important implications for school leaders. Synthesising and disseminating these vital messages will form the backbone of a new phase of EEF work beginning later in the year.’

It will be interesting to monitor the impact of this work on the communication of outcomes from these particular evaluations.

It will be important to ensure that synthesis and dissemination is not at the expense of accuracy, particularly when ‘high stakes’ results are involved, otherwise there is a risk that users will lose faith in the independence of EEF and its willingness to ‘speak truth unto power’.

.

Maths Mastery Press Release

By also releasing their own posts on 13 February, Mathematics Mastery and Ark made sure that they too would not be picked up by the media.

They must have concluded that, even if they placed the most positive interpretation on the outcomes, they would find it hard to create the kind of media coverage that would generate increased demand from schools.

The Mathematics Mastery release – ‘Mathematics Mastery speeds up pupils’ progress – and is value for money too’ – begins with a list of bullet points citing other evidence that the programme works, so implying that the EEF evaluations are relatively insignificant additions to this comprehensive evidence base:

  • ‘Headteachers say that the teaching of mathematics in their schools has improved
  • Headteachers are happy to recommend us to other schools
  • Numerous Ofsted inspections have praised the “new approach to mathematics” in partner schools
  • Extremely positive evaluations of our training and our school development visits
  • We have an exceptionally high retention rate – schools want to continue in the partnership
  • Great Key Stage 1 results in a large number of schools.’

Much of this is hearsay, or else vague reference to quantitative evidence that is not published openly.

The optimistic comment on the EEF evaluations is:

‘We’re pleased with the finding that, looking at both our primary and secondary programmes together, pupils in the Mathematics Mastery schools make one month’s extra progress on average compared to pupils in the other schools after a one year “dose” of the programme…

…This is a really pleasing outcome – trials of this kind are very rigorous.  Over 80 primary schools and 50 secondary schools were involved in the testing, with over 4000 pupils involved in each phase.  Studies like this often don’t show any progress at all, particularly in the early years of implementation and if, like ours, the programme is aimed at all pupils and not just particular groups.  What’s more, because of the large sample size, the difference in scores between the Mathematics Mastery and other schools is “statistically significant” which means the results are very unlikely to be due to chance.’

The section I have emboldened is in stark contrast to the EEF blog post above, which has the title:

‘Today’s findings; impact, no-impact and inconclusive – a normal distribution of findings’

And so suggests exactly the opposite.

I have already shown just how borderline the calculation of ‘statistical significance’ has been.

The release concludes:

‘Of course we’re pleased with the extra progress even after a limited time, but we’re interested in long term change and long term development and improvement.  We’re determined to work with our partner schools to show what’s possible over pupils’ whole school careers…but it’s nice to know we’ve already started to succeed!’

 .

There was a single retweet of the Tweet above, but from a particularly authoritative source (who also sits on Ark’s Advisory Group).

.

Ark Press Release

Ark’s press release – ‘Independent evaluation shows Mathematics Mastery pupils doing better than their peers’ – is even more bullish.

The opening paragraph claims that:

‘A new independent report from the independent Education Endowment Foundation (EEF) demonstrates the success of the Mathematics Mastery programme. Carried out by academics from Cambridge University and the Institute of Education, the data indicates that the programme may have the potential to halve the attainment gap with high performing countries in the far East.

The second emboldened statement is particularly brazen since there is no evidence in either of the reports that would support such a claim. It is only true in the sense that any programme ‘may have the potential’ to achieve any particularly ambitious outcome.

Statistical significance is again celebrated, though it is important to give Ark credit for adding:

‘…but it is important to note that these individual studies did not reach the threshold for statistical significance. It is only at the combined level across 127 schools and 10,114 pupils that there are sufficient schools and statistical power to determine an effect size of 1 month overall.’

Even if this rather implies that the individual evaluations were somehow at fault for being too small and so not generating ‘sufficient statistical power’.

Then the release returns to its initial theme:

‘… According to the OECD, by age fifteen, pupils in Singapore, Japan, South Korea and China are three years ahead of pupils in England in mathematical achievement. Maths Mastery is inspired by the techniques and strategies used in these countries.

Because Maths Mastery is a whole school programme of training and a cumulative curriculum, rather than a catch-up intervention, this could be a sustained impact. A 2 month gain every primary year and 1 month gain every secondary year could see pupils more than one and a half years ahead by age 16 – halving the gap with higher performing jurisdictions.’

In other words, Ark extrapolates equivalent gains – eschewing all statistical hedging – for each year of study, adding them together to suggest a potential 18 month gain.

It also seems to apply the effect to all participants rather than to the average participant.

This must have been a step too far, even for Ark’s publicity machine.

.

maths mastery ark release capture

.

They subsequently changed the final paragraph above – which one can still find in the version within Google’s cache – to read:

‘…Because Maths Mastery is a whole school programme of training and a cumulative curriculum, rather than a catch-up intervention, we expect this to be a sustained impact.  A longer follow-up study will be needed to investigate this.’

Even in sacrificing the misleading quantification, they could not resist bumping up ‘this could be a sustained impact’ to ‘we expect this to be a sustained impact’

 .

[Postscript: On 25 February, Bank of America Merrill Lynch published a press release announcing a £750,000 donation to Maths Mastery.

The final paragraph ‘About Maths Mastery’ says:

‘Mathematics Mastery is an innovative maths teaching framework, supporting schools, students and teachers to be successful at maths. There are currently 192 Mathematics Mastery partner schools across England, reaching 34,800 pupils. Over the next five years the programme aims to expand to 500 schools, and reach 300,000 pupils. Maths Mastery was recently evaluated by the independent Education Endowment Foundation and pupils were found to be up to two months ahead of their peers in just the first year of the programme. Longer term, this could see pupils more than a year and a half ahead by age 16 – halving the gap with pupils in countries such as Japan, Singapore and China.’

This exemplifies perfectly how such questionable statements are repurposed and recycled with impunity. It is high time that the EEF published a code of practice to help ensure that the outcomes of its evaluations are not misrepresented.]  

.

Conclusion

.

 .

Representing the key findings

My best effort at a balanced presentation of these findings would include the key points below. I am happy to consider amendments, additions and improvements:

  • On average, pupils in primary schools adopting Mathematics Mastery made two months more progress than pupils in primary schools that did not. (This is a borderline result, in that it is only just above the score denoting one month’s progress. It falls to one month’s progress if the effect size is calculated to three decimal places.) The effect is classified as ‘Low’ and this outcome is not statistically significant. 
  • On average, pupils in secondary schools adopting Mathematics Mastery made one month more progress than pupils in secondary schools that did not. The effect is classified as ‘Low’ and this outcome is not statistically significant. 
  • When the results of the primary and secondary evaluations are combined through meta-analysis, pupils in schools adopting Maths Mastery made one month more progress than pupils in schools that did not. The effect is classified as ‘Low’. This outcome is marginally statistically significant, provided that the 95% confidence interval is calculated to three decimal places (but it is not statistically significant if calculated to two decimal places). Care is needed in analysing meta-analysed findings because… [add explanation]. 
  • There is relatively little evidence that the primary programme is more effective for learners with lower prior attainment, but there is such evidence for the secondary programme (in respect of non-calculator questions). There is no substantive evidence that the secondary programme has a different impact on pupils eligible for free schools meals. 
  • The per-pupil cost is relatively low, but the initial outlay of £6,000 for primary schools with 2FE and above is not inconsiderable. Mathematics Mastery may represent a cost-effective change for schools to consider. 
  • The evaluations assessed the impact of the programme in its first year of adoption. It is not appropriate to draw inferences from the findings above to attribute potential value to the whole programme. EEF will be evaluating the medium and long-term impact of the approach by [outline the methodology agreed].

In the meantime, it would be helpful for Ark and Maths Mastery to be much more transparent about KS1 assessment outcomes across their partner schools and possibly publish their own analysis based on comparison between schools undertaking the programme and matched control schools with similar intakes.

And it would be helpful for all partners to explain and evidence more fully the benefits to high attainers of the Maths Mastery approach – and to consider how it might be supplemented when it does not provide the blend of challenge and support that best meets their needs.

It is disappointing that, three years on, the failure of the National Curriculum Expert Panel to reconcile their advocacy for mastery with stretch and challenge for high attainers – in defiance of their remit to consider the latter as well as the former –  is being perpetuated across the system.

NCETM might usefully revisit their guidance on high attainers in primary schools to reflect their new-found commitment to mastery, while also incorporating additional material covering the point above.

.

Postscript

A summary of this piece, published by Schools Week, prompted two comments – one from Stephen Gorard, the other from Dylan Wiliam. The Twitter embed below is the record of a subsequent debate between us and some others, about design of the Maths Mastery evaluations, what they tell us and how useful they are, especially to policy makers.

One of the tweets contains a commitment on the part of Anna Vignoles to set up a seminar to discuss these issues further.

The widget stores the tweets in reverse order (most recent first). Scroll down to the bottom to follow the discussion in chronological order.

.

.

GP

February 2015

The Politics of Setting

I had been intending never to revisit the difficult topic of setting, secure in the knowledge that I could not improve on my earlier treatment of the pros and cons.

P1010978

Irrelevant picture of Norway by Gifted Phoenix

But recent developments have caused me to reconsider, led me to address the issue from a different perspective.

My previous post attempted an objective and balanced statement of the educational arguments for and against, drawing on the research evidence and taking account of all learners, regardless of their attainment.

This one explores how setting – just one option within the far wider range of so-called ‘ability grouping’ strategies – has been reflected in government policy and party political policy documents since 1997, culminating in the position we have  reached as the parties begin to firm up their 2015 General Election manifestos.

The post begins with brief notes on terminology and incidence.

The substantive text is organised into four sections:

  • How Labour government positions on setting developed and fluctuated between 1997 and 2010.
  • How the Conservative party turned to setting while in opposition.
  • How Coalition Government policy on setting has shifted since May 2010.

It concludes with a summary of the position we have reached as we approach the next Election, together with some ideas for how we might move forwards more constructively.

In case you prefer to read selectively I have included links to the relevant section from each of the bullet points above.

 

Terminology

I take setting to mean grouping learners in a discrete class, ostensibly selected with reference to prior attainment in a specific subject.

It is distinct from streaming, where the selection – which may well be generic and ability-based – applies to teaching across a range of different subjects. The learners in a higher stream may not be higher attainers in each of these subjects.

One sometimes also encounters references to banding which is broadly synonymous with streaming, except that streaming tends to refer to a single class, while bands may include more than one class. It may therefore be a less differentiated form of streaming.

Both setting and streaming are within-school selection strategies, which may be adopted by selective or comprehensive schools. They may be perceived as viable alternatives to between-school selection which is no longer regarded as politically feasible by Labour, Conservatives or Liberal Democrats.

There is, however, continuing pressure from the right of the Conservative party and recently from UKIP for the restoration of grammar schools. The Coalition government has opened up the prospect of satellite establishments and overseen the introduction of several selective post-16 institutions. This might be viewed as the thin end of the wedge.

It has not always been possible to isolate the approach to setting since there is often a tendency to brigade it with streaming and/or a wider range of grouping strategies, occasionally including various approaches to within class grouping.

Sometimes these distinctions are clear and sometimes they are less so. To take a prominent example, the relevant entry in the Education Endowment Foundation’s Teaching and Learning Toolkit is not a model of clarity.

Called ‘Setting or Streaming’, it discusses effectiveness initially in terms of ‘ability grouping’ (first paragraph).

Clarity is not improved by the inclusion of the American terms for streaming (tracking) and setting (regrouping).

Nor is it clear whether ‘ability grouping’ is intended as a synonym for ‘setting or streaming’ or whether it has a broader scope.

The second paragraph reverts to ‘setting or streaming’ before discussing a wider range of interventions targeted at gifted and talented learners including several accelerative measures. One of these – promotion – is not necessarily a grouping strategy, at least as I understand the term.

The next three paragraphs relate to low attainers. The third focuses on ‘ability grouping’, although there is one reference is to ‘setting or streaming’, the fourth discusses both ‘setting’ and ‘ability grouping’, while the fifth mentions only ‘ability grouping’.

This terminological imprecision is confusing and unhelpful, especially when it appears in a text that purports to present the available research evidence clearly and unambiguously.

 

How prevalent is setting?

There are few recent and reliable statistics available on the incidence of setting.

Statistics deposited in the Commons Library in March 2012 (Dep 2012-0434) provide Ofsted data on the percentage of lessons observed in secondary schools that were either setted or streamed/banded for every year from 1996/97 to 2002/03, excluding PE.

In 2002/03, 40% of all secondary lessons observed were setted and 4% were streamed or banded.

From 2003/04 to 2010/11, the table provides percentages of lessons observed that were setted, streamed or banded, for ‘lower’, ‘average’ and ‘upper ability’ learners respectively.

In 2010/11, the average percentages across all year groups were 12% for average ability, 16% for lower ability and 17% for higher ability.

The reply to a PQ from July 2011 provides 2009/10 data, for maths, English and science in primary and secondary schools respectively. The percentages relate to ‘classes setted, streamed or banded by ability where pupils are placed within an ability range within the school’.

The average figures across all year groups are set out below. For primary schools I have included Year 6 percentages in brackets:

  • Maths primary 19% (34%)
  • English primary 11% (19%)
  • Science primary 2% (3%)
  • Maths secondary 71%
  • English secondary 58%
  • Science secondary 62%

A 2014 study of primary practice found that:

Approximately 17% of the pupils studied, who were born in 2000-2001, were in ability streams. Some 8% of the total group were in the top stream, 5% in the middle and 4% in the bottom stream.

Last year Ofsted estimated that, excluding PE, some 45% of secondary lessons were set or streamed. The TES story containing these figures notes:

‘The Department for Education was unable to produce statistics on how many students are set or streamed. Ofsted produced limited data based on lessons it had inspected… but stressed that “there is no way of using this data to draw out national conclusions in any way”….

…In comments accompanying Ofsted’s figures, Sir Michael noted that, since 2005, its inspections have not involved observing all teachers in a school. Lessons that were seen were not “necessarily representative” of the school or system as a whole, he said.

….”It is not possible to deduce from inspection data the proportions of pupils nationally who are taught in setted/streamed classes or in mixed-ability groups,” the chief inspector said.’

We can only conclude that a significant proportion of secondary students and older primary learners is setted and that that this practice is most prevalent in the core subjects. It is unclear whether these percentages are now increasing, stable or declining. 

It would be highly desirable to obtain more accurate figures through the School Census, if only to track the influence of the presentation of the evidence base in the Toolkit.

 

Part One: The evolution of Labour government policy from 1997 to 2010

 

First Labour Government

In 1997 the incoming Labour Government published its White Paper ‘Excellence in Schools’. The chapter on ‘Modernising the comprehensive principle’ said:

Mixed ability grouping… requires excellent teaching and in some schools has worked well. But in too many cases it has failed both to stretch the brightest and to respond to the needs of those who have fallen behind. Setting, particularly in science, maths and languages, is proving effective in many schools. We do not believe that any single model of grouping pupils should be imposed on secondary schools, but unless a school can demonstrate that it is getting better than expected results through a different approach, we do make the presumption that setting should be the norm in secondary schools. In some cases, it is worth considering in primary schools. Schools should make clear in reports to parents the use they are making of different grouping approaches. OFSTED inspections will also report on this.

The clear implication is that, where the quality of teaching is not excellent, setting is likely to prove relatively more effective than ‘mixed ability grouping’, particularly in science, maths and languages.

Setting will not be made compulsory in secondary schools, but there is a presumption that it should be ‘the norm’, presumably in all subjects but certainly in science, maths and languages, unless schools can show ‘better than expected results’ through a different approach. In primary schools, setting should be considered in some unspecified cases.

Ofsted will check what schools are doing (and presumably validate or otherwise any claim of ‘better than expected results’, although the precise meaning of this term is not explained).

The text also says that the Department will publish guidance and exemplification of best practice, taken from this country and abroad ‘in organising classes to meet the different abilities of pupils’. There is a list of strategies in which it has particular interest, including:

  • ‘target-grouping, where pupils are grouped by ability for part of the week and groups are altered in line with regular assessment;
  • fast-tracking, where pupils are encouraged to learn and take qualifications ahead of their age cohort.’

Early in 1999, Ofsted published a survey on setting in primary schools. I cannot source the text online, but contemporary reviews, such as this from the LGA, show that it was strongly supportive of the practice:

Setting, rather than streaming, in primary schools provides a powerful lever for raising standards, so long as it is carefully implemented and properly managed, say Her Majesty’s Inspectors from OFSTED.

A new survey of the practice of setting – grouping children by ability for specific subjects – uses evidence from OFSTED inspection data, from a questionnaire and from focused inspections by HMI. It endorses the government’s view that setting is well worth considering.

‘Where teachers understand its potential and modify their teaching techniques accordingly, setting can be a very successful way of organising teaching groups,’ HMI say in the report Setting in Primary Schools, published today by OFSTED.

They point out that setting does not, by itself, guarantee success in raising standards nor can it compensate for poor teaching. However, evidence from school inspections suggests that the quality of teaching in setted lessons in the three core subjects is slightly better than in lessons with the full ability range.’

This introduces two important themes – that the efficacy of setting is dependent on:

  • it being implemented and managed effectively and
  • the appropriate adaptation of teaching techniques.

In September 2000 a DfEE research report on ‘Innovative Grouping Practices in Secondary Schools’ summarises the advantages and disadvantages of ability grouping more generally, but consists mainly of extended case studies of contemporary innovative practice.

The introduction sets the context thus:

The challenge now is to find ways of grouping pupils and developing pedagogy that capitalises on the advantages and minimises the disadvantages outlined above. In other words, how can schools develop grouping plans to achieve the best attainment outcomes for pupils while minimising any negative impact?

This rather more pragmatic approach reappears in subsequent guidance documents, but was set aside when government policy was articulated.

 

Second Labour Government

The 2001 Green Paper ‘Schools Building on Success’ reverts to a bullish reference to setting in the section on KS3:

We want to see further increases in the extent of setting within subjects including express sets to enable those who are capable of doing so to advance beyond the levels set for their age and to take Key Stage 3 tests early.’

But this does not survive into ‘Schools Achieving Success’, the White Paper published the same year, which makes no reference to setting specifically or ‘ability grouping’ more generally..

A roughly contemporary PQ reply also hedges its bets:

The Government supports a flexible approach to pupil grouping, including setting by ability where appropriate’.

The sentence is vacuous because deliberately imprecise. Essentially it expresses the government’s preference for schools to decide their own approaches.

It seems that there is growing indecision over which line to take. Should the government opt for consistent and wholehearted endorsement, full devolution of responsibility to schools, or a middle path that focuses on developing and disseminating effective practice to meet the needs of different settings?

This is of course redolent of wider contemporary debate about the role of the government in determining education policy and practice.

Setting is not mentioned in the ‘Five Year Strategy for Children and Learners’ which appeared in 2004.

 

Third Labour Government

Setting makes a significant reappearance in the October 2005 White Paper ‘Higher Standards, Better Schools For All’:

‘Grouping students can help to build motivation, social skills and independence; and most importantly can raise standards because pupils are better engaged in their own learning. We have encouraged schools to use setting since 1997. Putting children in different ability groups within a class is commonplace in primary schools. Ofsted reports show that the proportion of Key Stage 3 lessons which are set has risen since 1997 to over a third now, with greater rises in English and maths. The significant majority of English, science and modern foreign language lessons in secondary schools, and about nine in ten maths lessons are already organised by setting.

It will continue to be for schools to decide how and when to group and set by ability. But we will encourage more schools to adopt such grouping and help them to learn from the innovative practices that some schools are already employing without lowering expectations for pupils in lower ability groups or limiting choices in the curriculum. We will publish, in the New Year, independent research into current best practice.

The first emboldened point implies a consistency that is not fully reflected in the narrative above, in that the encouragement for setting seems to have waned somewhat between 2001 and 2004.

The second emboldened section makes it clear that schools remain free to determine their own approaches. The presumption in favour of setting has gone by the wayside and the government will focus instead on encouragement through the continuing promotion of innovation and best practice.

Shortly afterwards, the research report ‘The Effects of Pupil Grouping: Literature Review’ appeared.

Back in 2010 I summarised its key findings thus:

  • No single form of grouping benefits all pupils and there is little attainment advantage associated with setting – ie no significant difference between setting and mixed ability classes in overall attainment outcomes across all pupils.
  • ‘At the extremes of attainment’ low-achieving pupils show more progress in mixed ability classes and high-achieving pupils show more progress in sets.
  • Lower sets tend to contain a disproportionate number of boys, pupils from those ethnic groups that tend to underachieve and pupils with SEN.
  • There are aspirational and behavioural disadvantages to setting, predominantly amongst lower attainers, and there is a correlation between disaffection and setting, particular for pupils in the lowest sets.
  • Higher sets are more likely to have experienced and highly-qualified teachers whereas lower sets experience more changes of teacher and are less likely to be taught by a specialist in the subject.’

A contemporaneous TES story argues that the report undermines the government’s position by offering too little support for setting:

‘Setting pupils by ability, one of the most widely-trailed parts of last week’s white paper, has few benefits, a study funded by the Department for Education and Skills has concluded.

There is no evidence that streamed or set classes produce, on average, higher performance than mixed-ability classes, said the report. It also found that setting pupils is already widespread, particularly in maths….

It says the debate between setting and mixed-ability teaching has become polarised and does not reflect what happens in schools where a wide range of ways of grouping pupils is used….

…The review concluded: “There are no significant differences between setting and mixed-ability teaching in overall attainment … but … low-achieving pupils show more progress in mixed-ability classes and high-achieving pupils show more progress in set classes.’

This provides the spur for a renewed effort to push beyond the polarised debate, to refocus on helping to develop solutions to fit particular needs and circumstances

In 2006, DfES published ‘Pupil Grouping Strategies and Practices at Key Stage 2 and 3: Case Studies of 24 Schools in England’, a companion piece to the 2005 study.

The impact of grouping on pupil attainment were summarised thus:

  • Schools identified that the use of setting enabled them to tailor teaching for different ability pupils in order to impact on their understanding and achievement. However, the research did not find evidence to corroborate these expected achievement gains.
  • In secondary schools that adopted mixed ability or part mixed ability grouping approaches, the rationale given by teachers and senior managers tended not to make reference to attainment but rather to focus on the benefits in terms of social awareness and inclusivity. 
  • In primary schools, which used mixed ability as the predominant organisational grouping, pupils were often seated around tables on the basis of ability and it was not possible to differentiate attainment outcomes that related directly to setting or mixed ability from these observations.’

So advocates of secondary setting could not demonstrate stronger attainment overall, while advocates of secondary mixed ability teaching were not primarily concerned with the impact on attainment.

In the primary sector it was not possible to distinguish a differential impact on outcomes from either option.

In September of the same year, the National Strategies produced ‘Grouping Pupils for Success’, useful guidance for schools deciding on the most appropriate grouping strategies.

The introduction says that it:

‘…moves on from the old ‘for and against’ debates about grouping to a more sophisticated understanding of what it means to group pupils for success.’

Suggestions relating specifically to setting include:

  • ‘Make a careful match of individual teacher strengths with the nature of sets, for example placing a teacher experienced in challenging low attainers with the lowest set or band, to lift attainment.
  • Avoid ‘teaching to the middle’ in mixed-ability classes.
  • Monitor pupils’ learning to ensure that pupils have opportunities to demonstrate higher attainment, for example in tiered papers in the National Curriculum tests, and that access to the curriculum and resources are not limited by assumptions about ability level.
  • Ensure that teaching in top sets creates a learning atmosphere in which it is acceptable to make mistakes, to ask for clarification or repetition.
  • Develop inclusive teaching approaches, for example through differentiated questioning or the use of within-class groupings.’

It summarises the research on setting and mixed ability grouping respectively in the two tables reproduced below.

 

2014 Setting Capture 1

2014 setting Capture 2

 

Effective Teaching and Learning for Pupils in Low Attaining Groups’ (2007) takes the same line as the previous studies in arguing that:

‘…the polarisation of the grouping debate does not reflect the available evidence….Rather than pointing towards the overwhelming superiority of one form of grouping over another, it suggests that different forms of grouping are effective for different ‘types’ of pupils, in relation to different kinds of outcomes.’

But it continues:

‘The decision, therefore, about whether to group by attainment, either has to be seen as a matter of principle, where empirical evidence is of limited relevance, or else has to be regarded as one that is complex and may even be too close to call.

Nevertheless, the authors contribute some further empirical evidence to the debate, notably concerning the characteristics of pupils in low attaining sets:

  • ‘The analysis of data on pupils’ allocation to groups confirms prior attainment as the main, albeit a relatively poor predictor of set placement, for example, with over half the pupils with low prior attainment in English ending up in middle or high sets. Although prior attainment remains statistically significant, setting decisions are clearly not made on this basis alone.’
  • ‘Social class is a significant predictor of set placement. Pupils from higher socio-economic status (SES) backgrounds are more likely to be assigned to higher sets and less likely to be assigned to lower sets.
  • Special Educational Need (SEN) is a significant predictor of set placement (after controlling for social class and prior attainment), with these pupils concentrated in the low attainment sets. Less than 10% of pupils in the highest sets have SEN. This suggests that SEN and low attainment are seen as closely related or overlapping and that set placement may also be confounded by the effect of behaviour.
  • Ethnicity was a weaker significant predictor of set placement, (after controlling for social class and prior attainment), with pupils of Bangladeshi origin being slightly less likely to be selected for the higher sets.
  • Gender was not a significant predictor of set placement (after controlling for social class and prior attainment), except in Key Stage 2 literacy where, against recent trends, females were more likely to be placed in a low set. Overall, males are slightly overrepresented in the low sets and under-represented in the middle sets but this difference was not statistically significant.
  • Other factors including teacher assessments, teacher judgements and pupil characteristics such as behaviour are likely to influence set placement. Some schools allocated pupils with behavioural difficulties to high sets irrespective of prior attainment because they believed that the classroom context provided in these groups would promote positive behaviour. Other schools allocated these pupils to lower sets because they were smaller and provided higher staff ratios.’

Also in 2007, ‘The Children’s Plan’ included a section entitled ‘Good classroom practices – better use of grouping and setting’.

Essentially this replicates the approach taken in the 2005 White Paper, though the drafting is far more convoluted and so far less clear:

‘Improved understanding of each child’s progress should also lead to more effective use of group teaching. Since 1997 we have been encouraging schools to use ‘setting’ (teaching groups of pupils by ability in a particular subject rather than across a range of subjects) and other forms of pupil grouping, and we continue to encourage these practices.

Using setting and groups to teach children of similar abilities and interests can bring real educational benefits. But where it is poorly implemented, for example through ‘streaming’ (where pupils are grouped across a range of subjects based on general rather than subject-specific assessment) it can be socially divisive and detrimental to all but the highest achieving pupils. Grouping can also be used more effectively in the classroom – in particular, through proven approaches to in-class grouping by need, and guided group work when the teacher coaches a small group to apply immediately what they have been learning in the main part of the lesson. We will promote this best practice as standard practice.

Under this new formulation, there is recognition that there can be effective practice in setting and mixed ability grouping alike.

The final sentence potentially embodies a slightly different approach, by introducing the notion of promoting ‘standard practice’, but no further details are provided about what exactly this will entail.

Then Labour appears to lose interest in setting. A 2008 publication from DCSF ‘Personalised Learning: A Practical Guide’ includes a chapter on ‘Pupil Grouping’ but it says almost nothing about setting. It is as if the authors are keen to move on from what has become a rather sterile debate.

A PQ from March 2009 uses the Children’s Plan formulation:

‘Analysis of research suggests that no single model of pupil grouping will be of benefit to all pupils all of the time. For example, there is some evidence that being taught in a mixed ability class can be beneficial for low attainers, but that ability-based classes can be beneficial for high attainers.

We promote setting — the grouping of pupils according to their ability in a particular subject — as an effective way of ensuring that individual pupils are receiving personalised help appropriate to where they are in their learning. Similarly, we promote effective pupil grouping practices, and guided work, as tools for delivering the most appropriate curriculum to each individual in mixed ability classes.

We do not promote streaming—where pupils are assigned to classes on the basis of an overall assessment of their general ability and pupils remain in their streamed classes across the majority of subjects—as it assumes that children will have the same level of ability in all subjects.’

But, three months later, the 2009 White Paper ‘Your child, your schools, our future’ has nothing to say on the subject, and references to setting are conspicuously absent from the Pupil and Parent Guarantees. Ministers have apparently decided that schools are best left to their own devices.

Setting did not appear in Labour’s 2010 Election Manifesto either.

 

Part 2: The Conservatives in opposition

In January 2006, just at the time when Government research reports were discussing the polarised nature of debate on setting and advocating a more nuanced approach, David Cameron and David Willetts (then Tory education spokesman) both made statements that explicitly supported setting.

One report has Cameron saying:

‘I want no child held back, so my priority is not selection by ability between schools but setting by ability within schools, because every parent knows that a high quality education means engaging children at the right level.’

Another attributes to him the statement:

‘I want the Conservative Party to help me campaign in setting by each subject in every school so that we actually do what I think is common sense which is to help stretch the brightest pupils and help those who are in danger of falling behind…There’s a real case for more selection within schools rather than selection between schools.’

“The government is getting into a mess over the issues of selection and admissions.”

It seems that Cameron has identified support for setting as a means of distancing himself from calls within his party for the introduction of more selective schools.

Willetts said:

What I shall be looking for in the months ahead is how best to spread setting, and I would not rule out using central government more in this area…The evidence that setting works is powerful indeed, and yet you still have more than half of lessons not taught in sets, where you can target your teaching methods to children with a particular level of skill.’

Another report has a slightly different version:

We are not saying that an edict will go out from the Department for Education that schools are instructed to set in all circumstances but the empirical evidence is that it works.

I would not rule out ministers getting involved in the way schools organise setting, but our instincts are to cut back rather than add to central bureaucracy and direction.

One can see writ large the tension between the mutually exclusive desires for prescription and autonomy. Willetts is leaving open the possibility of central direction of some sort.

In the event, it was decided that Ofsted would be the enforcer. The November 2007 Conservative Green Paper ‘Raising the Bar, Closing the Gap’ made this clear:

‘While every pupil must be given the opportunity of a good education, we also recognise that each pupil should be given the opportunity to learn in accordance with their particular aptitude and ability, so that the brightest pupils continue to be stretched at the same time as pupils who might be struggling are given extra support.

We believe that setting by ability is the only solution to achieving this ambition. Labour’s 1997 manifesto acknowledged the importance of setting and implied that the amount of setting in schools would be increased significantly. This has not taken place.

… We believe that school children learn more effectively when taught with children of a similar ability. We also believe setting contributes to better behaviour. We will therefore alter guidance to Ofsted to ensure that schools – particularly those not performing at high levels – set all academic subjects by ability.’

Contemporary press reports remind us that Cameron had originally spoken of ‘a grammar school stream’ in every school, but streaming was set aside in favour of setting:

‘The Tories will now make clear that streaming need only apply in smaller schools with limited timetabling.’

Cameron has decided that setting is ‘the only solution’ and the inspection regime will impose this on all schools (the document is not clear whether primary schools are included). There is no explicit exemption for those ‘performing at high levels’ although they will be a lower priority.

This new position is a restatement of the Labour position of 1997.

Hansard shows that new opposition spokesman Michael Gove maintained an interest in the issue until at least summer 2009.

In May 2008 he requests the latest data on the extent of setting and DCSF’s guidance on the issue. Ofsted answers the first point while the reply directs him to the materials referenced above.

In July 2009 he again asks for updated data on the incidence of setting.

But the enforcement of setting through Ofsted was clearly set aside when the time came to consider the content of the 2010 Tory Election Manifesto.

For the time being at least, it seemed that the alternative attractions of full autonomy for schools had triumphed.

 

Part 3: The evolution of Coalition policy on setting

 

2010 to 2014

Indeed, the 2010 Schools White Paper made a virtue of schools’ autonomy in such matters.

‘We will expect schools to set their own improvement priorities. As long as schools provide a good education, we will not mandate specific approaches.

We…believe that it is often effective to incentivise improvement and innovative ideas, rather than to mandate a uniform approach.’

But it takes some time for any evidence of this approach in relation to setting.

A PQ from July 2011 asking for data and current guidance to schools elicits the statement that, while there is no guidance,

Case studies showing the effective use of setting in schools will be made available on the department’s website shortly.’

The task was not a high priority. Eight months later, in March 2012, the answer to the next PQ on the topic confirms that the material has now been published.

These case studies were not transferred to gov.uk, but have been archived and are still available.

The covering article, dated 26 April 2012, reads:

Setting and other forms of pupil grouping are ways of tailoring teaching and learning for mixed-ability classes which can help raise standards.  When setting is done well it can be an effective way to personalise teaching and learning to the differing needs of groups of pupils.’

There are five case studies in all, two of secondary and three of primary schools. Each is no more than a page in length. Compared with some of the guidance produced by Labour administrations they are of relatively limited value.

But the issue was soon stirred up by the intervention of HMCI Wilshaw.

In September 2012, he referred to the issue obliquely, but in such a way that his comments could be interpreted to fit the very different political perspectives of the newspapers that carried his comments.

I can find no official Ofsted record of what he said.

One report offers this version:

‘Heads have got to make up their mind. If they want mixed-ability, then they have got to make sure there’s differentiated teaching. And we will be very critical when we inspect schools, particularly in the secondary sector, if we see mixed-ability without mixed-ability teaching.

He added: ‘If you have got a youngster with low basic skills sitting alongside a youngster with Oxbridge potential, then it is really important that that’s taken into account.’

Another provides a slightly different interpretation:

‘”Where there are mixed-ability classes, unless there is differentiated teaching… it doesn’t work,” he said, adding that effective differentiated teaching was “hugely difficult” to achieve.

He said mixed-ability classes could be an “article of faith” for schools who were not concerned enough about good practice and were doing something “more akin to social engineering”. In those cases Ofsted inspections would be “very critical”.’

A third suggests that Wilshaw expressly put some distance between his remarks and the setting controversy:

This is not a judgment on mixed ability as opposed to setting or streaming, it is saying where there are mixed ability classes unless there is differentiated teaching to groups of school children in the class, unless there are individual programmes of work, it doesn’t work,” he said,

“It is absolutely critical that if you have a youngster with low grades at school who struggles with literacy and numeracy sitting alongside a youngster with Oxbridge potential then it is really important that is taken into account and they are taught by people who are experienced in good teaching of mixed ability classes.”’

Conversely, a fourth was much more bullish:

‘Inspectors will now be critical of schools that do not differentiate between high and low achievers.

This could lead to schools falling into the new category of ‘requires improvement’ (which replaces the old ‘satisfactory’ description), or even being labelled ‘inadequate’…

Ofsted cannot force schools to adopt setting – grouping pupils according to their academic ability in single subjects – or streaming, where ability groups cover most or all subjects.

However, Sir Michael’s intervention is likely to make headteachers rethink their practice of mixed ability classes for fear of being marked down in future inspections

‘It’s a combination of low expectations of what these youngsters can achieve, that their progress is not sufficiently tracked, and what I would call and have done ever since I have been a teacher the curse of mixed-ability classes without mixed-ability teaching,’ he said.

The former head said mixed-ability classes did not work ‘unless there is differentiated teaching to groups of schoolchildren in the class’ and ‘individual programmes of work’….

…Many schools had recognised this and ‘moved towards setting arrangements’, he said.

It seems as though everyone heard what they wanted to hear.

Wilshaw’s fundamental point seems to echo the 1997 White Paper and some of Labour’s guidance material reviewed above.

His principal argument is that mixed ability settings require mixed ability teaching, and that effective mixed ability teaching is a difficult skill to master. He implies, but does not state explicitly, that teaching a narrower range of ability is comparatively easier.

He suggests that Ofsted will look askance at schools that adopt mixed ability teaching on ideological grounds,  that cannot justify it in terms of their learners’ achievement, or where the quality of teaching is insufficient to support it.

Seven months later, in April 2013, a peculiar story appeared in the TES called ‘Conservatives abandon pledge to enforce ability grouping’:

‘The practice of grouping classes by ability has long had strong backing from the top. Ofsted, the education secretary, the prime minister and their Labour predecessors have all encouraged schools to use setting in more lessons.

But, despite their rhetoric, Conservative ministers have quietly dropped a pledge to enforce setting by ability…

… Last September, Ofsted chief inspector Sir Michael Wilshaw appeared to support the call, warning that some students were being held back by “the curse of mixed-ability classes without mixed-ability teaching”, adding that such teaching was “hugely difficult” to achieve.

But the government has now said that it does not advocate setting. “It is for schools to decide how best to organise teaching – including whether to group and set pupils by ability – as they know exactly what their students need,” a spokesman said.

And Ofsted says it “doesn’t have a view on whether setting or streaming is a good idea or not”. A spokeswoman for the inspectorate also revealed that Conservative ministers had not asked Ofsted to enforce setting.’

This is odd, since any Conservative adherence to the enforcement of setting would date back to 2007. I can find no more recent commitment than that. So why overtly drop a policy that no-one could reasonably have assumed the Conservatives still to advocate?

It is as if ministers are determined to re-impose the position on autonomy reached back in 2010, which has been compromised by Wilshaw’s insistence on linking schools’ decisions to Ofsted’s assessment of their performance.

Given more recent history, it is also conceivable that ministers were on the receiving end of pressure from the Prime Minister’s Office to adopt a more interventionist approach. Perhaps this was their way of distancing education ministers from such pressure.

But Ofsted’s alleged neutrality on the question of setting was soon called into question again when in June 2013 it published  ‘The Most Able Students’.

This developed HMCI Wilshaw’s theme:

‘In around a third of the schools visited, students were taught mainly in mixed ability groups throughout Key Stage 3. Where setting by ability occurred at an early stage, this was usually only for mathematics. Sets were introduced at various times for English and science, but often only in the later stages of Key Stage 3.

For most other subjects, mixed ability classes were retained throughout Key Stage 3. In the very best schools, this did not appear to have a detrimental impact on students’ progress because the teaching was carefully planned and well matched to the most able students’ needs. In the less effective schools, the work was pitched at the level of the average-attaining students. It was not challenging enough for the most able and their progress was insufficient…

…It was evident in some of the schools visited that school leaders had responded to recent research findings about mixed ability teaching, particularly in Key Stage 3. Eight of the schools had moved recently towards grouping by ability, particularly in English, mathematics and science. Some other school leaders recognised that their earlier grouping arrangements had not always promoted the best outcomes for the most able students. They indicated that they were moving away from mixed ability teaching to setting, streaming or banding in most subjects. Schools’ data shown to inspectors during the visits indicated that these moves were beginning to have a positive impact on outcomes for the most able students.’ 

Although Ofsted may not have an official view on the desirability of setting, it is abundantly clear that schools are encouraged to consider it where the quality of mixed ability teaching provided is not sufficiently strong to secure commensurate outcomes for able learners.

The current iteration of the inspection handbook says:

Inspectors should consider how effectively pupils are grouped within lessons and across year groups. For example:

  • where pupils are taught in mixed ability groups/classes, inspectors will consider whether the most able are stretched and the least able are supported sufficiently to reach their full potential. 
  • where pupils are taught in sets, inspectors will consider how leaders ensure that pupils in lower sets are not disadvantaged or that teachers take into account that pupils within a set may still have very different needs. ‘

The associated grade descriptors for an inadequate school mention:

‘The organisation of the curriculum and classes is resulting in some pupils achieving less well than they should.’

This carefully balanced approach makes it clear that inspectors will consider equally seriously the efficacy of sets and mixed ability groups.

Schools are likely to be pulled up if their mixed ability settings are insufficiently challenging for high attainers, but will also be challenged if their sets are holding back low attainers or if teaching within sets is insufficiently differentiated.

 

Developments in Autumn 2014

This careful balance was once more disturbed.

On 3 September, the Guardian reported that:

‘Compulsory setting according to ability in England’s secondary schools is to be proposed by the education secretary, Nicky Morgan, in her first big initiative since she took the role in July. She is due to make the announcement as early as today.’

This shock introduction was immediately undermined by a subsequent paragraph indicating that setting would not be compulsory after all, though it would be incentivised through the inspection regime:

‘It is expected that Morgan will ask the education watchdog, Ofsted, to implement and enforce the measure, probably by making it a condition of receiving an outstanding rating‘.

So the strategy would be to prevent schools from receiving the top inspection rating if they had not adopted setting.

The piece also stated unequivocally that the policy had been ‘cleared with Downing Street’, although ‘The Department for Education gave no comment after being contacted.’

This implied that the source of the story was the Prime Minister’s Office.

Just six hours later, the same paper carried a second report featuring comments made by Morgan in a Commons debate that same afternoon:

‘Richard Fuller: The Secretary of State has faced a number of confusing interventions from Opposition Members, one of which repeated something that was said in The Guardian today, which was that she was about to announce a policy a compulsory setting. Will she take this opportunity to say whether she is going to do that?

Nicky Morgan: Let me confirm for the benefit of the House that there is absolutely no truth in those rumours. There are some people outside this House who have a rather unhealthy interest in speculating about what I am or am not about to announce. They would be better served if they spent less time on Twitter and talking to journalists, and more time reflecting on the importance of the policies and reforms that have already been implemented by this Government.’ (Hansard, 3 Sep 2014, Col 357)

So as not to appear entirely wrong-footed, the Guardian cited Dominic Cummings in support of its original story:

Gove’s former special adviser Dominic Cummings said he had been told Cameron wanted to back compulsory setting.

He added on Twitter: “I was told by No 10 and two others in Whitehall a version v close to the Guardian story. Some had warned internally it was mad.” He also suggested there was a launch plan prepared inside No 10.’

Cummings’ Twitter feed on the day in question is instructive:

 

 

A BBC report included comment from the Lib Dems, confirming that they would not support a Coalition policy along these lines:

‘A senior Liberal Democrat source also distanced the party from any such proposal.

“This has not been agreed by the Liberal Democrats and is not government policy. We do not think it would be appropriate to tie schools’ hands in this way.”’

And Labour in opposition took a similar line:

‘Labour’s shadow education secretary Tristram Hunt had called on the education secretary to reject political involvement in such school decisions.

“I believe that excellent heads and great teachers know better than Westminster politicians how to deliver the best schooling for all pupils.

“We thought there was political consensus on the importance of school autonomy.’

The Cummings version lends support to the idea that some sort of enforcement of setting remains under consideration for inclusion in the Conservative Election Manifesto for 2015.

It might once again help to pacify those in the Party who seek a renewed commitment to selective education. Conservative MPs will be acutely aware of UKIP’s declared policy:

‘Existing schools will be allowed to apply to become grammar schools and select according to ability and aptitude. Selection ages will be flexible and determined by the school in consultation with the local authority.’

I could find no explicit statement to the effect that a commitment to introduce setting would definitely not be in the 2015 Manifesto. The final paragraph of a related TES story claimed this was the case, but this is not supported elsewhere.

While there was no reference to setting in Morgan’s speech to the Conservative Party Conference, the idea has subsequently reappeared in a different guise.

On 12 October the Conservative party let it be known that their Manifesto would include plans to enable Regional Schools Commissioners to intervene directly in the operation of any school rated inadequate by Ofsted, whether or not an academy.

The briefing made an explicit link with setting:

‘A Conservative spokesperson said the new powers would be developed in “consultation with Ofsted and the Education Endowment Foundation”, but a “menu of options” might include forcing schools to put children into classes based on ability, or ‘sets’ as they are also known.’  (Academies Week).

So, rather than making setting a condition of an ‘outstanding’ Ofsted rating, this possible new approach is to empower RSCs to impose setting on inadequate schools.

Whether the inclusion of setting in the menu of options would survive the consultation process is open to question – and presumably RSCs would also be reluctant to impose it without hard evidence that it would radically improve the performance of an inadequate school. Such evidence would be hard to find.

Perhaps this is a method of parking the issue:  giving No 10 the impression that enforcement of setting is part of the agenda when in fact it is not.

Meanwhile, the DfE has restated its existing commitment to giving schools autonomy in this matter. On 30 October, a Conservative MP tabled a PQ:

‘Andrew Rosindell (Romford):

To ask the Secretary of State for Education, what steps her Department is taking to ensure that children at secondary school are being efficiently grouped according to their academic ability.

Answered by: Mr David Laws

The Department for Education believes that individual schools are best placed to determine whether and how to group children by academic ability. There are many different models of pupil grouping, and schools themselves are best able to respond to their individual circumstances to meet the needs and capabilities of their pupils.

Note that the reply refers to the DfE’s belief rather than the Government’s position.

This suggests that we may not have heard the last of the matter, especially if setting remains part of the Prime Minister’s strategy for buying off the siren voices calling for renewed commitment to grammar schools.

 

Part 4: The Education Endowment Foundation’s Evidence Base

The Education Endowment Foundation (EEF) exists to improve the achievement of disadvantaged learners. The website says:

‘We aim to raise the attainment of children facing disadvantage by:

  • Identifying and funding promising educational innovations that address the needs of disadvantaged children in primary and secondary schools in England;
  • Evaluating these innovations to extend and secure the evidence on what works and can be made to work at scale;
  • Encouraging schools, government, charities, and others to apply evidence and adopt innovations found to be effective.’

But, confusingly, it has also been designated jointly with the Sutton Trust as a What Works Centre for improving educational outcomes for all school age children:

‘The What Works centres will summarise and share research with local decision-makers, helping them to invest in services that deliver the best outcomes for citizens and value-for-money for taxpayers.

In the EEF’s case, decision-makers include teachers and school-leaders, parents and governors, researchers and policy-makers. They are the primary audience for our Teaching and Learning Toolkit, an accessible summary of educational research which provides guidance for teachers and schools on how to use their resources to improve the attainment of disadvantaged pupils. ‘

See the logical disconnect? The principal tool used by the EEF/Sutton Trust to inform decision makers about what works well with all learners has been designed to inform decisions about what works well with disadvantaged learners.

This is particularly problematic when it comes to setting.

 

 The Teaching and Learning Toolkit

The EEF’s website describes the Toolkit as follows:

‘The Sutton Trust-EEF Teaching and Learning Toolkit is an accessible summary of educational research which provides guidance for teachers and schools on how to use their resources to improve the attainment of disadvantaged pupils.

The Toolkit currently covers 34 topics, each summarised in terms of their average impact on attainment, the strength of the evidence supporting them and their cost.’

One of the 34 topics is ‘Setting or streaming’. This pairing is potentially problematic since the subsequent commentary does not consistently distinguish the impact of one from the other.

I have already described above how the guidance switches between setting, streaming, ability grouping and wider gifted and talented provision.

When it comes to quantification, the Toolkit arrives at an average impact measure of -1 month – ie in terms of average pupil progress over a year, the impact of ‘setting or streaming’ on disadvantaged learners is negative.

The description of the Toolkit notes:

‘Most approaches included in the Toolkit tend to have very similar average impacts on pupils with different characteristics. However, where the research summarised suggests that an approach has a different average impact on the learning of pupils from disadvantaged backgrounds compared to the learning of their peers, the Toolkit’s ‘headline’ average impact figure refers to the former.’

The section describing the impact of ‘setting or streaming’ begins:

Overall, ability grouping appears to benefit higher attaining pupils and be detrimental to the learning of mid-range and lower attaining learners. On average, ability grouping does not appear to be an effective strategy for raising the attainment of disadvantaged pupils, who are more likely to be assigned to lower groups.’

It continues:

On average, studies show that higher attaining learners make between one and two additional months progress when set or streamed compared to when taught in mixed ability groups.’

No reference is made to the plight of disadvantaged high attainers, who might be expected to benefit commensurately.

The impact of setting and streaming remains undifferentiated.

The next section of the commentary considers a wider range of grouping interventions targeted on gifted and talented learners. This does not seem directly relevant to the narrower case of ‘setting or streaming’.

The final section of the commentary is concerned with low attainers (and so, by implication, includes the majority of disadvantaged learners).

It says:

Low attaining learners fall behind by one or two months a year, on average, when compared with the progress of similar students in classes without ability grouping. It appears likely that routine setting or streaming arrangements undermine low attainers’ confidence and discourage the belief that attainment can be improved through effort. Research also suggests that ability grouping can have a longer term negative effect on the attitudes and engagement of low attaining pupils. It should be noted that there are some exceptions to this average, where ability grouping has benefitted all learners. Further study could be undertaken to understand what happened differently in these examples.

Evidence suggests that the impact of setting is more detrimental to low attaining pupils in mathematics who do better in mixed attainment groups, and that ability grouping particularly affects upper primary and lower secondary education. The effects appear to be less clear-cut in other subjects, though negative effects are reported for low attaining pupils across the curriculum.

Though the average impact of ability grouping on low attaining pupils is negative, evidence suggests that certain types of ability grouping are more effective than others. Some studies have shown that reducing the size of the lowest attaining groups and assigning high-performing teachers to these groups can be effective, as can providing additional targeted catch up support.’

So the text suggests that high attaining learners make between one and two months more progress in sets/streams, while low attaining learners fall behind by the same amount. There is, therefore, between two and four months’ difference in the impact on high and low attainers respectively.

But this commentary:

  • Is not providing sufficiently accurate information to enable us to distinguish the impact of setting alone, as opposed to ‘setting or streaming’ together. 
  • Neglects the interests of high-attaining disadvantaged learners who are assumed to be an insignificant minority. 
  • Is fundamentally unclear, a particularly heinous crime considering the purpose of the Toolkit.

 

The KCL Study

One of the projects funded by the EEF is examining Best Practice in Grouping Students. The four-year project began in 2014 and continues through to spring 2018. It is co-ordinated by a team based at King’s College London and is receiving a total of £1.184m. The evaluation has been assigned to the NFER.

The project summary on EEF’s website distinguishes two parallel strands:

  • A randomised control trial of an intervention ‘which trains schools in a best practice approach to setting’ and is focused on English and maths in Years 7 and 8. The trial begins in September 2015, but results are not expected until spring 2018. It will be conducted in a sample of 120 schools, randomly allocated to conduct the trial or form part of the control group.
  • A pilot of an intervention ‘to introduce mixed ability teaching to secondary schools’ and ‘examine whether it is possible to overcome the common barriers to mixed ability teaching’. The intervention will be developed initially with three schools but the pilot will subsequently be extended to ten. Results are due in spring 2017.

One of the descriptions on the King’s College site suggests that the focus is explicitly placed on lower sets and low attainers:

The project addresses the needs of pupils in low ‘ability’ sets and streams, wherein research has identified socially disadvantaged pupils are strongly over-represented.

The project draws on substantial existing research evidence (concerning the educational outcomes for young people in low sets and streams, and the related poor practice often associated with such groupings), as illustrated in the Education Endowment Foundation/Sutton Trust Toolkit and elsewhere. The evidence from the literature concerning existing bad practice and detrimental outcomes associated with low ‘ability’ groups establishes areas for potential improvement, which will be applied via the interventions.’

It adds that the trial will measure the impact on pupil attainment, noting that the developmental phase:

‘Will also allow us to research why schools and policy-makers appear so wedded to ‘ability’ grouping, and what might persuade the adoption of an evidence-based approach.’

A second description confirms the focus of the setting strand on lower sets:

One, on Best Practice in Setting, seeks to remedy the detrimental elements identified by research to be associated with low sets.’

This also mentions that a pilot study – it is not clear whether this is of one strand or both – is being undertaken in six schools in the current academic year. It states that the full study will involve around 100 schools (rather than 120) and will be completed in 2017 (rather than spring 2018).

The exclusive emphasis on low sets is directly contradicted in a TES story about the project:

The Education Endowment Foundation, which commissioned the study, said the research aimed to address poor practices in both low and high sets.’

Is there a difference of opinion between KCL and the EEF? It would be helpful to know the truth, since there is otherwise strong reason to believe that the needs of high-attaining disadvantaged learners will be neglected.

NFER’s description of its evaluation currently says that the protocols for both strands are not yet agreed. Hopefully they willl be clear on whether the operation of higher sets – and the impact on disadvantaged learners within them – is also part of the agenda.

 

What Makes Great Teaching?

On 31 October 2014, the Sutton Trust published ‘What Makes Great Teaching?

One assumes that it has done so in its role as partner with the EEF in the What Works Centre, rather than as a charity supporting social mobility.

The press release dictated much of the media coverage. Provocatively headed:

‘Many popular teaching practices are ineffective, warns new Sutton Trust report’

It begins:

‘Lavish praise for students is among seven popular teaching practices not supported by evidence, according to a new Sutton Trust report which reviews over 200 pieces of research on how to develop great teachers.

What Makes Great Teaching, by Professor Rob Coe and colleagues at Durham University, warns that many common practices can be harmful to learning and have no grounding in research. Examples include using praise lavishly, allowing learners to discover key ideas by themselves, grouping students by ability and presenting information to students based on their “preferred learning style”.’

Later on the press release lists seven ‘examples of strategies unsupported by evidence’. Third in the list is:

Grouping students by ability. Evidence on the effects of grouping by ability, either by allocating students to different classes, or to within-class groups, suggests that it makes very little difference to learning outcomes. It can result in teachers failing to accommodate different needs within an ability group and over-playing differences between groups, going too fast with the high-ability groups and too slow with the low.’

The Report itself does not include this list in its executive summary. It appears in a section called ‘Examples of Ineffective Practices’. But the text repeats more or less verbatim the claim in the press release

The following are examples of practices whose use is not supported by the research evidence…

Group learners by ability

Evidence on the effects of grouping by ability, either by allocating students to different classes, or to within-class groups, suggests that it makes very little difference to learning outcomes (Higgins et al, 2014). Although ability grouping can in theory allow teachers to target a narrower range of pace and content of lessons, it can also create an exaggerated sense of within-group homogeneity and between-group heterogeneity in the teacher’s mind (Stipek, 2010). This can result in teachers failing to make necessary accommodations for the range of different needs within a supposedly homogeneous ‘ability’ group, and over-doing their accommodations for different groups, going too fast with the high-ability groups and too slow with the low.’

The first reference to grouping by ability making ‘very little reference to learning outcomes’ is to the Toolkit, though the report’s bibliography attributes it to 2013 not 2014. The second reference – ‘Stipek 2010’ – inexplicably appears in the bibliography under D rather than S.

As far as I can see, this is a reference to an article – an excerpt from a 2002 book called Motivation to Learn: Integrating Theory and Practice – that cites a series of other studies dating between 1976 and 1998.

Is the opening sentence an accurate description of what the Toolkit says?

As we have seen, the Toolkit considers ‘setting or streaming’ – though it also mentions a range of other strategies targeted at gifted and talented students – but it doesn’t discuss substantive evidence relating to within-class groups.

The only reference to them comes at the end of the Toolkit entry, in the section ‘What should I consider’. It says:

‘Flexible within-class grouping is preferable to tracking or streaming for low-attaining pupils’.

But that doesn’t support the statement above. (Nor, for that matter, is it supported by the evidence in the earlier parts of the Toolkit text.)’

The differential impact of setting on high and low attainers is not mentioned.

How might this statement be improved to reflect the evidence? It might say:

  • When discussing evidence on the effectiveness of ability grouping, it is important to distinguish the impact of setting, streaming and within class ability grouping respectively.
  • It is also important to distinguish the differential impacts on high attainers and low attainers respectively. Great care should be taken to clarify whether the discussion relates to all learners or only to disadvantaged learners. The subset of low attainers ought not to be regarded as analogous with the subset of disadvantaged learners.
  • The evidence suggests the overall impact of setting or streaming – ie one or the other – on low attainers is negative (one to two months) whereas the impact on high attainers is positive (one to two months). There is therefore a difference of up to four months’ progress between high and low attainers respectively.
  • There is less evidence on the differential impact of setting and streaming respectively. What we do know is x.
  • The impact of setting varies according to prior attainment of the learners, the subject of study and how well it is implemented. The available evidence suggests that setting is most likely to be successful under the following conditions….and, conversely, is least likely to be successful when….

 

Where we are now – and future prospects

 

The Evidence Base

The arguments about the advantages and disadvantages of setting have long been polarised – and there is little evidence to suggest that this will change as we head into 2015.

The EEF/Sutton Trust nexus purports to stand for evidence-based pedagogy, but both arms of the partnership are too vague and unspecific in how they present this evidence.

Because they take short cuts, it is too easy to interpret their coverage as overwhelmingly negative towards setting. A more careful and nuanced presentation would highlight the different contexts where setting might be more and less likely to operate effectively.

As things stand, the standard bearers for evidence-based practice seem more inclined to perpetuate the polarisation of views, rather than promoting a more sophisticated understanding of the issue.

This may be a deliberate reaction to the unevidenced espousal of setting by politicians, or it may just be insufficient attention to detail.

Substantive amendment of the toolkit entry – along the lines set out above – is devoutly to be wished for.

And it should be accompanied by a commitment to produce and update accurate data about the incidence of setting by sector, type of school, subject and pupils’ prior attainment. The Schools Census is the perfect vehicle.

One hopes that the results from the KCL study will be more carefully presented, but the absence of evaluation protocols and the disagreements over the focus of the study are a cause for concern. The protocols should be finalised and published forthwith.

The KCL study is unlikely to reveal that best practice in setting has a substantial impact on improvements in the performance of disadvantaged learners, even the high attainers.

But everything is relative: hardly any of the studies of other interventions so far completed by the EEF have identified a significant positive effect.

I would be satisfied with confirmation of a limited but positive impact on the performance of disadvantaged high attainers, combined with recognition that any negative impact on disadvantaged low attainers can potentially be eliminated through effective practice.

Some recommendations for the implementation of hybrid approaches – perhaps combining a single top set with several parallel mixed ability groups – wouldn’t go amiss.

Any evidence that does emerge from the KCL study – positive or negative – will not appear until well after the 2015 Election.

For the future, we anticipate keenly the pronouncements on setting that will emerge from a College of Teaching and/or from the Liberal Democrat’s independent Education Standards Authority. There is no reason to believe that they will be any more inclined to withstand the ideological pressures than their predecessors.

 

The Policies

Labour and Liberal Democrat politicians seem wedded to the notion of full autonomy for schools, though their parallel enthusiasm for the new entities mentioned above might tend to undermine this position.

It is not clear whether schools would be protected as much from the setting-related pronouncements of a College of Teaching as they would from the predilections of a government minister.

As for the Conservatives, they seem caught on the horns of a dilemma. Do they too opt for autonomy and the virtues of the market, or do they get on the front foot and develop a more robust alternative to UKIP’s espousal of selection?

They could commit to more selective schools, if only of the post-16 variety. They might even push back the concept to 14+, perhaps enabling the strongest sixth form colleges to accept 14-16 year-olds just as FE colleges can.

They might develop a new cross-school support system for high attainers, especially disadvantaged high attainers. They need look no further than posts elsewhere on this blog for ideas as to how it might be constructed.

They should avoid at all costs the Sutton Trust open access wheeze, which directs a substantial taxpayer subsidy towards independent schools while denuding the state sector of high attaining learners.

Or they might continue to refine the idea of a grammar stream in every school.

The research evidence against streaming seems to me more damning than the evidence against setting, though this is often obscured by the tendency to roll them up together.

That said, several comprehensive schools operating in direct competition with grammar schools seem to have introduced a grammar stream. It would be possible to promote this practice in this subset of comprehensive schools – whether through Ofsted or otherwise – and to develop practical guidance on effective practice.

Setting might well remain the path of least resistance, although compulsory setting across the board would be too much of a straitjacket, restricting the flexibility of those schools that perform outstandingly with mixed ability teaching.

So some sort of selective imposition is necessary. The Ofsted inspection regime is the only effective lever remaining in the hands of central government. The Inspection Handbook might be altered to reinstate a presumption of the kind advanced by Labour in 1997 – and this might be weighted towards schools in either the higher or the lower performance categories. In either case the presumption might be confined to the core subjects.

But even this could only be carried forward in the teeth of opposition from the education profession, so would have the potential to reduce still further the quantum of Tory teacher votes.

The more recently suggested fall-back – adding setting to a menu of possible school improvement interventions managed through the Regional Schools Commissioners – is weak by comparison. So weak that it is tantamount to kicking the whole idea into the long grass.

There are precious few alternatives. Perhaps the only other realistic proposition is to revert to presentation of the evidence base, but to develop this into substantive guidance that schools are expected to consider before determining their approach to ability grouping – much more substantive than the half-hearted case studies published in 2012.

If it were my call, I would construct a ‘flexible framework’ quality standard that defines the broad parameters of effective practice while permitting schools significant flexibility over interpretation of those in practice.

This would align with and reflect the available research evidence on the most effective approaches to setting, including advice on when setting is most likely to work and when it might be preferable to select an alternative strategy.

I would incorporate the standard into supplementary guidance for Ofsted inspectors, to ensure that inspection judgements on setting are fully consistent with it.

And I would strongly encourage schools to use it within their own improvement planning processes – and through peer-to-peer assessments undertaken by teaching schools, national leaders of education and other elements of the emerging self-improving schools system.

I would combine this with a national support programme for disadvantaged high attainers in Years 7-13, generously funded through matched topslices from the Pupil Premium and HE fair access budgets.

 

The Politics

With May and Johnson already on manoeuvres, not to mention continued pressure from the Brady/Davis camp, Cameron may be even more inclined to press ahead, even in the teeth of opposition from some DfE ministers.

In June 2013, polling suggested  (p8) that 43% of all voters – and 66% of Tory voters – agreed that:

‘The Government should encourage more schools to select by academic ability and build more grammar schools’.

It seems highly likely that, without any viable alternative, the Tories will haemorrhage right wing votes to UKIP over this issue. But there is a trade-off with teacher votes. Are they capable of defining an acceptable middle way, or are they doomed to fall between two stools?

They might at any rate consider some of the ideas set out above.

 

GP

November 2014