The Sutton Trust has published a fascinating study of the value an aptitude test (the SAT Reasoning TestTM) as a predictor of academic success. The report is of critical interest to anyone interested in access to higher education and access to graduate professions, such as the law. It is particularly interesting in the context of current debates about the value of aptitude tests being considered by both the Bar and the Law Society. Looked at 2,754 students graduating in 2009 and it is fair to say the Sutton Trust were hoping that aptitude test might be a vehicle for improving access into higher education for disadvantaged groups. Similar hopes have been voiced with regard to aptitude tests for the profession.
The primary aim of the study was:
“to examine whether the addition of the SAT® alongside A levels is better able to predict HE participation and outcomes than A levels alone. Two specific issues were also to be addressed, namely:
“Can the SAT® identify students with the potential to benefit from higher education whose ability is not adequately reflected in their A level results because of their (economically or educationally) disadvantaged circumstances?
“Can the SAT® distinguish helpfully between the most able applicants who get straight A grades at A level?”
The Study provides detailed, multi-level modelling of data on 2,754 students. It examines their A-level, O-level attainment, their performance using SAT® Reading, Writing and Maths tests and data on their schools (types of school, average performance of those schools). In relation to these particular tests they conclude:
“There is no evidence that the SAT® provides sufficient information to identify students with the potential to benefit from higher education whose ability is not adequately reflected in their prior attainment.”
“The SAT® does not distinguish helpfully between the most able applicants who get three or more A grades at A level. The SAT® Reading and Writing components do add some predictive power for some classes of degree at highly selective universities2, but add very little beyond the information provided by prior attainment, in particular prior attainment at GCSE.”
In other words, SATs add little to GCSEs and A-Level grades even at the margins. There is more troubling news for proponents of aptitude tests (if results for SATs are typical), they may prejudice particular groups:
“Regression analyses showed that female students, some ethnic minorities, students with special educational needs (SEN) and students learning English as an additional language (EAL) appeared to perform less well on the SAT® than would be expected from their GCSE and A level attainment.”
There are some interesting points of detail which Universities and firms, who sometimes look at A-level scores should pay attention to. In particular:
- average A-level and average GCSE attainment is better than total A-level point at predicting attainment at university.
- Male and female students do equally well when prior attainment is taken into account (and we know that female students do far better on undergraduate law courses)
- Students attending more selective Universities tend to be less likely to achieve as high a class of degree as students from less selective universities with similar attainment.
This last point is worth dwelling on:
“In almost all subject areas (14 out of 17), the Sutton Trust ‘Top 30’ universities actually gave out proportionally more first class degrees within our sample than the ‘other’ universities, so the UNIdiff finding is not simply due to fewer firsts being awarded at these highly selective universities. It suggests that due to the large number of very able students competing for first class honours, it is more difficult to obtain this classification in highly selective universities than in less selective institutions…. ….To what extent students are already aware of this when applying to universities is unclear and whether they are nevertheless prepared to join such highly competitive environments due to the ‘market value’ of the degrees they obtain when they graduate.”
This finding has echoes for me of US research looking at the importance of ‘prestige universities’ as a predictor of success in the professions.
Given the current debate about the influence of socio-economic class on entry to the profession, the research finds that State school pupils do significantly better than public (or grammar school) pupils when they enter higher education than is predicted by their A-level grades. One way of putting this is that A-level grades exaggerate the abilities of public school students. At least part of the ‘leg up’ that such students receive as a result of the school they attend disappears during University education.
“…although independent and grammar school students disproportionately go to more selective universities they still perform less well than their peers. In one of the models reported later (section 5.1 and model 19 in appendix 3), students from the most highly selective universities (the Sutton Trust Top 30) were excluded, yet the grammar school and independent school coefficients were still significant. In other words students from independent and grammar schools are performing below expectations in other universities, not just the highly selective ones.”
The results suggest that students from state school with the same A-level grades as those from public schools are, in terms of University performance, likely to perform better. Universities, firms and Chambers should be bearing this in mind when they admit students or recruit trainees and pupil barristers. The report writers put it like this: “a comprehensive student with grades BBB is likely to perform as well at university as an independent or grammar school student with grades ABB or AAB.” It is not uncommon to hear recruiters say that public school students get the benefit of better education and that this endures in their performance. These results suggest not.
The importance of thinking about the context in which students have been educated is also very important given the tendency of the profession to recruit from elite institutions. The research indicates that it is harder to get a 2:1 or 1 from these institutions but also that, “those in independent schools tended to achieve places on more prestigious courses”. Their tendency to achieve such places at prestige Universities is an effect independent of their attainment: that is a student with three AAAs from a state school is less likely to go to a prestigious institution than someone with three AAAs from a public school (even though that state student is more likely to get a 2:1 or 1st!). State school pupils are less likely to apply. Furthermore, “More affluent students were more likely to be studying on courses with high entry point requirements than might be expected from their attainment.”
This research is a fascinating indication of what a detailed look at how well selection of students works. The results suggest educators and professional recruiters genuinely interested in selecting the best candidates should pay close attention to school performance and/or school type. In particular student performance has to be placed in a context which better predicts likely performance. One possibility is that students from State Schools should benefit from more generous entry requirements than students from Grammar and Independent Schools if the best students are to be recruited.
It is important to note that although SATs did not appear to predict future performance in this context, other aptitude tests may prove useful in the future in particular contexts. As the report writers note:
“Despite considerable variation in SAT® scores between HE applicants with similar A level attainment, this variation has not proved to be useful in predicting degree outcomes. Other aptitude or admissions tests may also differentiate between applicants in terms of test scores but such differentiation may not be helpful in predicting undergraduate success. Aptitude tests or other admissions tests may well be useful in contexts where other attainment data is lacking. However, the use of any admission test needs to be validated; supported by evidence that it is a reliable and valid measure of HE performance.”
It will be interesting to see how rigorous the Bar’s current testing of aptitude tests is and also how the Law Society decide to proceed in the light of their current investigations. The Sutton Trust Report shows the benefits of proper testing. In the context of University admissions, I would say the view of this report is pretty clear: these aptitude tests are a waste of money.