Measurement error distorts inequality of opportunities in education

Inequality of opportunities in education is a major societal problem. But contrary to what many people think, inequality of opportunities is barely amplified by the track recommendations teachers issue in primary education. It is known that children with lowly-educated parents receive lower track recommendations than children of highly-educated parents, even if they have the same scores on the final test in 6th grade.

"This is often seen as proof of bias among teachers. But this conclusion can only be drawn if the final test is a perfect measure to capture students' capacities and talents, and that's not the case," Thomas van Huizen (Utrecht University School of Economics) states.

The same score for the final test, different track recommendations

Various reports from organisations such as the Dutch Inspectorate of Education and the CPB Netherlands Bureau for Economic Policy Analysis show that children with lowly-educated parents relatively often receive lower track recommendations than the secondary track recommendations according to the final test (the test advice). This underrecommendation can indicate inequality of opportunities. But the problem is that ‘the same score on the final test, different track recommendations’ is often wrongly interpreted as ‘the same capacities, different track recommendations’.

"People think much too quickly that there has to be a bias among teachers. That they must be having lower expectations from children from underprivileged environments. But we can't draw that conclusion based on the existing results. So we really have to be more careful with that," says Van Huizen.

"There are roughly three possible explanations for the often-found result of ‘the same score on the final test, different track recommendations,'" Van Huizen explains:
 

• First, the teacher is biased and therefore has the tendency to give children from underprivileged environments lower assessments. This possible explanation has received the most attention in the public and political discussions.
   
• Second, teachers include work attitude and other relevant skills alongside objective test results when drafting a track recommendation. That is also the intent in our education system.
   
• Third, the final test, like any test, has a measurement error or ‘noise’. The final test score actually measures a combination of capacities and measurement error. What do we mean with that? Of course, a final test remains an indication. So you can have a bad day, some bad luck – or good luck instead. On top of that: the test has a limited duration, which can just turn badly for you because of the selection and composition of the test questions. Or you can guess many answers correctly or incorrectly. After all, a test score is in part determined by all kinds of coincidental circumstances.

“The bigger the measurement error, the more inequality of opportunities is distorted. This explanation means that inequality of opportunities due to track recommendations is in part a ‘statistical artefact’ in any case. It's remarkable that this final argument is rarely mentioned in the discussion. The assumption is probably that this can't be an important explanation for the differences in track recommendations,” Van Huizen says.

Measurement error may be the most important explanation

But his new study instead shows that the measurement error is a very important explanation – maybe even the most important one – for ‘the same score for the final test, different track recommendations.’ If no corrections are made to correct for this measurement error, we do indeed see that children of highly-educated parents receive recommendations for havo or vwo more often than children of lowly-educated parents with the same score for the final test. “But after correcting for the measurement error, those differences cease to be. So measurement error plays a very big role. That's striking.”

Distortion caused by final test after track recommendation

"For a long time, the final test came first and the track recommendation only after that. That was usually consistent with each other," Van Huizen continues. "Since 2014/2015, the teacher first gives a track recommendation, which is guiding. The final test is taken after that, and that may only be corrected upwards. That has all kinds of consequences.

If parents are already satisfied with the recommendation, the pupils really don't have to answer any question in the final test; you can still go to the school level that is in the recommendation. To other children, the final test is still a last resort. They sometimes go all out to get the level up. Both result in additional noise in the measurement, a distortion of the consistency in the children's capacities. So the question is in which regard the differences between track recommendations and final test results say something about inequality of opportunities.

In between 2013/2014 and 2014/2015, inequality of opportunities seems to have significantly increased according to the Dutch Inspectorate of Education, but this is distorted by the changes in behaviour caused by the change of policy in 2014/2015.

In the Pre-advices of the Koninklijke Vereniging voor Staatshuishoudkunde (KVS) , which will be presented to the Dutch Ministry of Social Affairs and Employment next Monday, Ron Diris (Leiden University) and Lex Borghans (Maastricht University) also show that the differences in scores and recommendations remain reasonably constant in the years after the change in 2014/2015. In the past, these researchers have also pointed out the fact that measurement error could be a relevant explanation for ‘the same score for the final test, different track recommendations.’

Teachers have a view on capacities thanks to the tracking system

You can correct for a measurement error if you are a researcher, but a teacher also does that in practice. Because the teacher has access to an entire series of objective test data from the pupil tracking system. That is why the teacher can see whether or not there is a coincidental outlier. In that case, the measurement error will have played a big role and the teacher accounts for that. So if a pupil has an outlier upwards or downwards for a final test, a teacher will say: “That deviates so much from the pattern I've seen in the past years that I'm not going to change my recommendation over it.”

Van Huizen would not advise to end the final tests: “A final test provides valuable information, but isn't by definition better than the track recommendation.” But he does point out the importance of the pupil tracking system: “‘The teacher sees the movie, the final test sees a photo’ has been said at the introduction of the order: track recommendation first, then the final test. I think that's true, too. Children take very many tests. Teachers look at the consistency in these. They see this objective data come by very often and there are then all kinds of protocols to process them into the track recommendation. I think that because of this, there is very little room for teachers' bias in the Netherlands. Biases are at play everywhere and always, but the question is whether or not they play such a big role in Dutch education as is often thought. Earlier surveys seem to quite overestimate this.”

What you shouldn't do, is saying: ‘Considering the same capacities of pupils, children with lower-educated parents will receive lower recommendations.’ We can't draw that conclusion. It also distracts from the bigger problems of inequality of opportunities.

“What you shouldn't do, is saying: ‘Considering the same capacities of pupils, children with lower-educated parents will receive lower recommendations.’ We just can't draw that conclusion from the surveys. It also distracts from the bigger problems of inequality of opportunities: that the skills of groups of pupils are very far apart. This is not just about cognitive skills such as language and numeracy in those cases, but also about behaviour and work attitude. These differences in skills are already very visible at the age of two years[WB1] and are still there at the end of primary school. In the end, that's the core of the problem.”

Fight against inequality of opportunities has to primarily be fought in other fields

“We know that children of higher-educated parents do better in school on average – we see that not in a single test but in all tests. And already before children enter nursery class, the differences in skills are prominent. That's why you must already start fighting inequality in preschool education,” says Van Huizen, who does research into that himself as a part of the EVENING Project.

“Inequality of opportunities in education is a big problem, but the role of the ‘biased teacher’ in the problem as a whole seems to be very limited. There are other, bigger problems at play. For instance, not all groups have equally easy access to preschool facilities. The Dutch education system is also strongly segregated and children who have educational disadvantages are more often in schools that provide lower qualities of education. On top of that, it's more difficult for these schools to recruit and retain good teachers. Another problem is that secondary schools more and more often provide only one education level in one location, which makes the transitions such as from vmbo to havo more difficult.

The real fight to increase equality of opportunities has to be fought in these fields. And in that fight, hard-working teachers are the vanguard.

It's of course good that there is attention in education for expectations and that teachers are made aware of possible unconscious biases. But I can imagine very well that teachers won't like it if they keep getting branded as biased, as if they are the big cause of inequality of opportunities. While we need the teacher very much in order to close the gaps in opportunities. That's why it's counter-productive to keep emphasising that teachers are systematically biased, especially because there's no real hard evidence of it.”

More information

Do you have any more questions about this article? If you do, please contact Thomas van Huizen: t.m.vanhuizen@uu.nl .

Read Thomas van Huizen's working paper, ‘Teacher bias or measurement error bias? Evidence from track recommendations.’ (pdf)