Switch from paper to computer tests caused science test scores to drop, study finds
A SWITCH from paper to computer-based testing could explain a significant drop in science performance among teenagers in Ireland, a study has claimed.
A leading academic has questioned the findings of the latest wave of the OECD's flagship Programme for International Assessment (Pisa) study, due to changes made to the methodology.
In 2015, Pisa was conducted on computer in 58 countries, while 14 others used a standard paper test. This was a significant departure from previous cycles, when all countries assessed their children using paper-based assessment.
The most recent survey of 15-year-olds found that the performance of Northern Ireland pupils in tests in science, maths and reading remained stable.
Young people in the Republic were found to be among the best at reading in the developed world.
There was a significant drop in science performance.
It is believed this was partly linked to the introduction of computer-based tests, which involve more complex scientific enquiry. More than half of students in Ireland had never completed such tests on a computer before.
In addition, their use of computers in school and for homework was found to be significantly less than students across OECD countries.
The Centre for Education Economics (CfEE) has today published research, led by Professor Jerrim from the UCL Institute of Education, questioning the 2015 findings.
Using data from the Republic, Germany and Sweden, the paper illustrated how the changes could have had a significant impact upon the results.
It found that pupils tended to perform worse when taking a computer rather than paper based test.
Prof Jerrim questioned whether the methodology the OECD has used to "adjust" for this problem had worked sufficiently well, and if results from the Pisa main study were truly comparable to previous cycles.
"Taking a test on computer is very different to the standard procedure of taking a test using paper and pencil. Yet the OECD has provided scant evidence on the impact this is likely to have had upon the PISA 2015 results," he said.
"Could this have driven some of the more surprising findings from the PISA 2015 study, such as Scotland's plummeting performance on reading and science compared to 2012, the significant drop in science performance in Ireland and Germany compared to 2012, or the significant decline in several East Asian countries mathematics scores? I certainly don't think we can currently rule out such possibilities."
James Croft, founder and chairman of CfEE, said it was vital that there was clarity around the methodology of the assessments "as governments clearly rely on them when setting education policy".
"We hope that by publishing this paper, governments across the world will carefully reflect upon how comparable the 2015 results are both to other countries and to those from previous PISA assessments," he said.