Running Out of Time: Leveraging Process Data to Identify Students Who May Benefit from Extended Time
Keywords:
Extended Time Accommodation, NAEP Assessment, Process Data, Machine Learning, Test-Taking Behavior,, Machine Learning,Equitable Accommodations.Abstract
This study explored the effectiveness of extended time (ET) accommodations in the 2017 NAEP Grade 8 Mathematics assessment to enhance educational equity. Analyzing NAEP process data through an XGBoost model, we examined if early interactions with assessment items could predict students’ likelihood of requiring ET by identifying those who received a timeout message. The findings revealed that 72% of students with disabilities (SWDs) granted ET did not use it fully, while about 24% of students lacking ET were still actively engaged when timed out, indicating a considerable unmet need for ET. The model demonstrated high accuracy and recall in predicting the necessity for ET based on early test behaviors, with minimal influence from background variables such as eligibility for free lunch, English Language Learner (ELL) status, and disability status. These results underscore the potential of utilizing early assessment behaviors as reliable predictors for ET needs, advocating for the integration of predictive models into digital testing systems. Such an approach could enable real-time analysis and adjustments, thereby promoting a fairer assessment process where all students have the opportunity to fully demonstrate their knowledge.
Downloads
References
Elliott, S. N., & Marquart, A. M. (2004). Extended Time as a Testing Accommodation: Its Effects and Perceived Consequences. Exceptional Children, 70(3), 349-367. https://doi.org/10.1177/001440290407000306
Alster, E. H. (1997). The Effects of Extended Time on Algebra Test Scores for College Students With and Without Learning Disabilities. Journal of Learning Disabilities, 30(2), 222-227. https://doi.org/10.1177/002221949703000210
Lovett B. J., Leja A. (2013). Students’ perceptions of testing accommodations: What we know, what we need to know, and why it matters. Journal of Applied School Psychology, 29, 72–89.
Bolt, S. E., & Thurlow, M. L. (2006). Item-level effects of the read-aloud accommodation for students with reading disabilities (Synthesis Report 65). National Center on Educational Outcomes, University of Minnesota. Retrieved from https://files.eric.ed.gov/fulltext/ED495897.pdf.
Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. 22nd acm sigkdd international conference on knowledge discovery and data mining, (pp. 785-794).
De Brey, C., Zhang, A., & Dillow, S. (2023). Digest of Education Statistics 2021 (NCES 2023-009). Washington, DC.: National Center for Education Statistics.
Fuchs, L. S., Fuchs, D., & Capizzi, A. M. (2005). Identifying appropriate test accommodations for students with learning disabilities. Focus on Exceptional Children, 37(6).
Goldstone, L., Ressa, V., Lazarus, S. S., Hinkle, A. R., & Rogers, C. (2021). Extended Time: Research (NCEO Accommodations Toolkit #6a). Retrieved from National Center on Educational Outcomes.: https://publications.ici.umn.edu/nceo/accommodations-toolkit/extended-time-research
Gregg, N., & Nelson, J. M. (2012). Meta-analysis on the effectiveness of extra time as a test accommodation for transitioning adolescents with learning disabilities: More questions than answers. Journal of Learning Disabilities, 45(2), 128–138. https://doi.org/10.1177/0022219409355484
Koretz, D., & Hamilton, L. S. (2006). Testing for accountability in K-12. Retrieved from https://www.rand.org/pubs/external_publications/EP20060030.html
Lee, D., Buzick, H., Sireci, S. G., Lee, M., & Laitusis, C. (2021). Embedded Accommodation and Accessibility Support Usage on a Computer-Based Statewide Achievement Test. Practical Assessment, Research & Evaluation, 26, 25.
Lee, Y. H., & Haberman, S. J. (2016). Investigating test-taking behaviors using timing and process data. International Journal of Testing, 16(3), 240-267.
Lewandowski, L., Cohen, J., & Lovett, B. J. (2013). Effects of extended time allotments on reading comprehension performance of college students with and without learning disabilities. Journal of Psychoeducational Assessment, 31(3), 326-336.
Lipnevich, A & Panadero, E. (2021). A review of feedback models and theories: descriptions, definitions, and conclusions. Frontiers in Education, 6 (2021), 10.3389/feduc.2021.720195
Lovett, B. J. (2020). Disability Identification and Educational Accommodations: Lessons From the 2019 Admissions Scandal. Educational Researcher, 49(2), 125-129. https://doi.org/10.3102/0013189X20902100
Lovett, B. J. (2010). Extended time testing accommodations for students with disabilities: Answers to five fundamental questions. Review of Educational Research, 80(4), 611–638.
Lu, Y., & Sireci, S. G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26(4), 29–37.
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., . Lee, S.-I. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2, 1, 2522-5839.
National Center for Education Statistics (2023, September). Process Data From the 2017 NAEP Grade 4 and Grade 8 Mathematics Assessment. https://www.nationsreportcard.gov/process_data/
Nogueira, F. (2014). Bayesian Optimization: Open source constrained global optimization tool for Python. Retrieved from https://github.com/fmfn/BayesianOptimization
Ofiesh, N., Mather, N., & Russell, A. (2005). Using speeded cognitive, reading, and academic measures to determine the need for extended test time among university students with learning disabilities. Journal of Psychoeducational Assessment, 23(1), 35–52. https://doi.org/10.1177/073428290502300103
Osman, A. ,E., A., Ahmed, A.,N., Chow, M.,F., Huang,Y.,F., & El-Shafie, A. (2021). Extreme
gradient boosting (Xgboost) model to predict the groundwater levels in Selangor, Malaysia. Ain Shams Engineering Journal, 12(2), 1545-1556. https://doi.org/10.1016/j.asej.2020.11.011.
Provasnik, S. (2021). Process data, the new frontier for assessment development: rich new soil or
a quixotic quest?. Large-scale Assessments in Education, 9(1), 1-17.
Sahin, E., K. (2020). Assessing the predictive capability of ensemble tree methods for landslide
susceptibility mapping using XGBoost, gradient boosting machine, and random forest. SN Applied Sciences, 2:1308. https://doi.org/10.1007/s42452-020-3060-1
Sireci S.G., Banda E., Wells C.S. (2018) Promoting Valid Assessment of Students with Disabilities and English Learners. In: Elliott S., Kettler R., Beddow P., Kurz A. (eds) Handbook of Accessible Instruction and Testing Practices. Springer, Cham. https://doi.org/10.1007/978-3-319-71126-3_15
Sireci, S. G., Scarpati, S. E., & Li , S. (2005). Test accommodations for students with disabilities: An analysis of the interaction hypothesis. Review of Educational Research, 75(4), 457–490.
Thurlow, M. L., Elliott, J. L., & Ysseldyke, J. E. (2003). Testing students with disabilities: Practical strategies for complying with district and state requirements. Thousand Oaks, CA: Corwin Press.
Stone E.A. & Cook L.L. (2018) Fair Testing and the Role of Accessibility. In: Elliott S., Kettler R., Beddow P., Kurz A. (Eds.) Handbook of Accessible Instruction and Testing Practices. Springer, Cham. https://doi.org/10.1007/978-3-319-71126-3_4
Wolf, M. K., Hanwook Y., Guzman-Orth, D., & Abedi, J. (2022). Investigating the Effects of Test Accommodations with Process Data for English Learners in a Mathematics Assessment, Educational Assessment, 27(1), 27-45, DOI: 10.1080/10627197.2021.1982693
