Exploring Test Taking Disengagement in the Context of PISA 2022: Evidence from Process Data

Abstract

Achievement tests are commonly used in education to evaluate students' academic performance and proficiency in specific subject areas. However, there is a major problem that threatens the validity of achievement test scores which is test-taking disengagement. Respondents provide answers that are inconsistent with their true ability level and can introduce construct irrelevant variance that threatens the validity of scores. This study examines test-taking disengagement in the context of PISA 2022 using process data to identify patterns of behavior that influence student performance. Three key indicators; response time, number of actions and self-reported effort, were used to examine engagement levels. Employing Latent Profile Analysis (LPA), distinct profiles of test-takers were identified, ranging from highly engaged to disengaged groups. Results indicate that disengagement, characterized by low self-reported effort, minimal interactions, and rapid responses, is associated with lower test performance, threatening the validity of scores. These findings highlight the significance of accounting for disengagement when interpreting the results of large-scale assessments. The implications were discussed in relation to the existing literature and recommendations for future research were provided to address identified gaps and extend the study's contributions.