Eye-tracking as a tool in process-oriented reading test validation
Keywords:
reading comprehension, assessment, validity, eye-trackingAbstract
The present paper addresses the continuous need for methodological reflection on how to validate
inferences made on the basis of test scores. Validation is a process that requires many lines of
evidence. In this article we discuss the potential of eye tracking methodology in process-oriented
reading test validation. Methodological considerations are highlighted and special significance is
placed on the importance of studying the first reading of a text as well as reading while answering
questions about it. This point of view expands the traditional scope of eye-tracking methodology in
reading research. We conducted a small-scale study in which 18 12-year olds read and answered
questions about a multimodal text. In this study comprehension scores were related to allocation of
visual attention in two conditions: (i) reading a text passage for the first time; and (ii) rereading of the
text passage while answering questions about it.
Downloads
Download data is not yet available.
References
Afflerbach, P. (2000). Verbal Reports and Protocol Analysis. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson
& R. Barr (Eds.), Handbook of Reading Research (vol. 3, pp. 163-179). Mahwah, New Jersey,
London: Lawrence Earlbaum Associates.
Afflerbach, P. & Johnston, P. (1984). Research methodology: On the use of verbal reports in reading
research. Journal of Reading Behaviour 16 (307-322).
Alderson, J. C. (2000). Assessing Reading. Cambridge: Cambridge University Press.
Allan, A. I. C. G. (1992). EFL reading comprehension test validation: investigating aspects of process
approaches. Unpublished PhD thesis, Lancaster University, Lancaster.
Andreassen, R. & Bråten, I. (2010). Examining the prediction of reading comprehension on different
multiple-choice tests. Journal of Research in Reading 33 (263-283).
Campbell, J.R. (2005). Single Instruments, Multiple Measures: Considering the Use of Multiple Item
Formats to Assess Reading Comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading
Comprehension and Assessment (pp. 347–369). Mahwah, New Jersey: Lawrence Earlbaum
Associates.
Cordon, L. A. & Day, J. D. (1996). Strategy use on standardized reading comprehension tests. Journal of
Educational Psychology 88 (288–295).
Ericsson, K. & Simon, K. (1984). Protocol Analysis: Verbal reports as data. Cambridge, MA; MIT Press.
Farr, R., Pritchard, R. & Smitten, B. (1990). A Description of What Happens When an Examinee Takes a
Multiple-Choice Reading Comprehension Test. Journal of Educational Measurement 27 (209–
226).
Goetz, E.T., Schallert, D. L., Reynolds, R. E. & Radin, D. J. (1983). Reading in perspective: What real cops
and pretend burglars look for in a story. Journal of Educational Psychology 75 (500–510).
Hannus, M. & Hyönä, J. (1999). Utilizations of Illustrations during Learning of Science Textbook
Passages Among Low- and High-Ability Children. Contemporary Educational Psychology 2 (95–
123).
Holmqvist, K., Holmberg, N., Holsanova, J., Tärning, J. & Engwall, B. (2006). Reading Information
Graphics – Eyetracking Studies with Experimental Conditions. In J. Errea (Ed.), Malofiej
Yearbook of Infographics (pp. 54–61). Society for News Design (SND-E), Navarra University,
Pamplona, Spain.
Holsanova, J., Holmberg, N. & Holmqvist, K. (2005). Tracing Integration of Text and Pictures in
Newspaper Reading. Lund University Cognitive Studies 125. Lund: Lund University.
Hyönä, J., Lorch, R. F. J., & Kaakinen, J. K. (2002). Individual Differences in Reading to Summarize
Expository Text: Evidence from Eye Fixation Patterns. Journal of Educational Psychology 94
(44–55).
Høien, T. & Tønnesen, G. (1998). Håndbok til Ordkjedetesten [Handbook for the word-chain test].
Stavanger: Stiftelsen Dysleksiforskning.
Johansen, E. B. & Steineger, E. (1999). Globus: Natur og miljøfag 7 [Globus: 7th year science and
environment studies’]. Oslo: J.W. Cappelen Forlag.
Johnson, D. & Kress, G. (2003). Globalisation, Literacy and Society: redesigning pedagogy and
assessment. Assessment in Education 10 (5-14).
Johnston, P. J. (1984). Assessment in Reading. In P.D. Pearson, R. Barr, M. Kamil and P. Mosenthal (Eds.),
Handbook of Reading Research (2nd ed., pp. 147–182). New York: Longman.
Just, M. A. & Carpenter, P. A. (1980). A theory of reading. From eye fixations to comprehension.
Psychological Review 87 (329-354).
Kaakinen, J.K. & Hyönä, J. (2005). Perspective Effects on Expository Text Comprehension: Evidence
from Think-Aloud Protocols, Eyetracking and Recall. Discourse Process 40 (239–257).
Kamil, M. L. (2004). The current state of quantitative research. Reading Research Quarterly 39 (100-107).
Kress, G. & van Leuven, T. (2001). Multimodal discourse. The modes and media of contemporary
communication. London: Arnold.
Land, M. & Tatler, B. (2009). Looking and Acting. Vision and Eye Movements in Natural Behaviour. Oxford,
New York: Oxford University Press.
Langer, J. (1987). The construction of meaning and the assessment of comprehension: An analysis of
reader performance on standardized test items. In R. O. Freedle & R. P. Duran (Eds.), Cognitive
and linguistic analyses of text performance (pp. 225-244). Norwood, NJ: Ablex.
Li, W. (1992). What is a test testing? An investigation of the agreement between students´ test taking
processes and test constructers´ presumptions. Unpublished MA thesis, Lancaster University,
Lancaster.
Messick, S. (1988). The Once and Future Issues of Validity: Assessing the Meaning and Consequences
of Measurement. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 33–46). Hillsdale, New
Jersey, Hove, London: Lawrence Earlbaum Associates.
Messick, S. (1995). Validity of Psychological Assessment: Validation of Inferences From Persons’
Responses and Performance as Scientific Inquiry Into Score Meaning. American Psychologist
50 (741–7499.
Mullis, I. V. S., Kennedy, A. M., Martin, M. O. & Sainsbury, M. (2006). PIRLS 2006 Assessment Framework
and Specifications. (2nd ed.). Chestnut Hill, MA: Boston College.
Ozuru, Y., Best, R., Bell, C., Witherspoon, A. & McNamara, D. S. (2007). Influence of Question Format and
Text Availability on the Assessment of Expository Text Comprehension. Cognition and
Instruction 25 (399-438).
Ozuru, Y., Dempsey, K., McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the
comprehension of science texts. Learning and Instruction 19 (228-242).
Paulson, E.J. & Henry, J. (2002). Does the Degrees of Reading Power Assessment Reflect the Reading
Process? An Eye-Movement Examination. Journal of Adolescent & Adult Literacy 46 (234–244).
Pearson, P. D. & Hamm, D. N. (2005). The Assessment of Reading Comprehension: A Review of
Practices – Past, Present and Future. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading
Comprehension and Assessment (pp. 13–69). Mahwah, New Jersey: Lawrence Erlbaum
Associates.
Raven, J.C. (1958). Standard Progressive Matrices: Sets A, B, C, D & E. Oxford, UK: Oxford Psychologists
Press Ltd.
Raven, J. C., Court, J. H. & Raven, J. (1988). Standard Progressive Matrices: 1988 Edition. London, UK:
Oxford Psychologists Press Ltd.
Rayner, K. (1992). Eye Movements and visual cognition: Scene perception during reading. New York:
Springer-Verlag.
Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research.
Psychological Bulletin 124 (372–422).
Rothkopf, E. Z. & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive
processing model from inspection times and eye movements. Journal of Educational
Psychology 71 (310–327).
Salmerón, L., Vidal-Abarca, E., Mana, A., Martínez, T., Gil, L. & Naumann, J. (submitted). Reading
Strategies in task oriented reading: The case of PISA-like tasks. Manuscript submitted for
publication.
Solheim, O. J. (2011). The Impact of Reading Self-Efficacy and Task Value on Reading Comprehension
Scores in Different Item Formats. Reading Psychology 32 (1-27).
Solheim, O. J. & Skaftun, A. (2009). The problem of semantic openness and constructed response.
Assessment in Education 16 (149-164).
Tai, R. H., Loehr, J. F. & Brigham, F. J. (2006). An exploration of the use of eye-gaze tracking to study
problem-solving on standardized science assessments. International Journal of Research &
Method in Education 29 (185–208).
Valencia, S. W. & Pearson, P. D. (1987). Reading Assessment: Time for a change. The Reading Teacher 40
(726–732).
Vidal-Abarca, E., Mãná, A. & Gil, L. (2010). Individual Differences for Self-Regulating Task-Oriented
Reading Activities. Journal of Educational Psychology 102 (817-826).
& R. Barr (Eds.), Handbook of Reading Research (vol. 3, pp. 163-179). Mahwah, New Jersey,
London: Lawrence Earlbaum Associates.
Afflerbach, P. & Johnston, P. (1984). Research methodology: On the use of verbal reports in reading
research. Journal of Reading Behaviour 16 (307-322).
Alderson, J. C. (2000). Assessing Reading. Cambridge: Cambridge University Press.
Allan, A. I. C. G. (1992). EFL reading comprehension test validation: investigating aspects of process
approaches. Unpublished PhD thesis, Lancaster University, Lancaster.
Andreassen, R. & Bråten, I. (2010). Examining the prediction of reading comprehension on different
multiple-choice tests. Journal of Research in Reading 33 (263-283).
Campbell, J.R. (2005). Single Instruments, Multiple Measures: Considering the Use of Multiple Item
Formats to Assess Reading Comprehension. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading
Comprehension and Assessment (pp. 347–369). Mahwah, New Jersey: Lawrence Earlbaum
Associates.
Cordon, L. A. & Day, J. D. (1996). Strategy use on standardized reading comprehension tests. Journal of
Educational Psychology 88 (288–295).
Ericsson, K. & Simon, K. (1984). Protocol Analysis: Verbal reports as data. Cambridge, MA; MIT Press.
Farr, R., Pritchard, R. & Smitten, B. (1990). A Description of What Happens When an Examinee Takes a
Multiple-Choice Reading Comprehension Test. Journal of Educational Measurement 27 (209–
226).
Goetz, E.T., Schallert, D. L., Reynolds, R. E. & Radin, D. J. (1983). Reading in perspective: What real cops
and pretend burglars look for in a story. Journal of Educational Psychology 75 (500–510).
Hannus, M. & Hyönä, J. (1999). Utilizations of Illustrations during Learning of Science Textbook
Passages Among Low- and High-Ability Children. Contemporary Educational Psychology 2 (95–
123).
Holmqvist, K., Holmberg, N., Holsanova, J., Tärning, J. & Engwall, B. (2006). Reading Information
Graphics – Eyetracking Studies with Experimental Conditions. In J. Errea (Ed.), Malofiej
Yearbook of Infographics (pp. 54–61). Society for News Design (SND-E), Navarra University,
Pamplona, Spain.
Holsanova, J., Holmberg, N. & Holmqvist, K. (2005). Tracing Integration of Text and Pictures in
Newspaper Reading. Lund University Cognitive Studies 125. Lund: Lund University.
Hyönä, J., Lorch, R. F. J., & Kaakinen, J. K. (2002). Individual Differences in Reading to Summarize
Expository Text: Evidence from Eye Fixation Patterns. Journal of Educational Psychology 94
(44–55).
Høien, T. & Tønnesen, G. (1998). Håndbok til Ordkjedetesten [Handbook for the word-chain test].
Stavanger: Stiftelsen Dysleksiforskning.
Johansen, E. B. & Steineger, E. (1999). Globus: Natur og miljøfag 7 [Globus: 7th year science and
environment studies’]. Oslo: J.W. Cappelen Forlag.
Johnson, D. & Kress, G. (2003). Globalisation, Literacy and Society: redesigning pedagogy and
assessment. Assessment in Education 10 (5-14).
Johnston, P. J. (1984). Assessment in Reading. In P.D. Pearson, R. Barr, M. Kamil and P. Mosenthal (Eds.),
Handbook of Reading Research (2nd ed., pp. 147–182). New York: Longman.
Just, M. A. & Carpenter, P. A. (1980). A theory of reading. From eye fixations to comprehension.
Psychological Review 87 (329-354).
Kaakinen, J.K. & Hyönä, J. (2005). Perspective Effects on Expository Text Comprehension: Evidence
from Think-Aloud Protocols, Eyetracking and Recall. Discourse Process 40 (239–257).
Kamil, M. L. (2004). The current state of quantitative research. Reading Research Quarterly 39 (100-107).
Kress, G. & van Leuven, T. (2001). Multimodal discourse. The modes and media of contemporary
communication. London: Arnold.
Land, M. & Tatler, B. (2009). Looking and Acting. Vision and Eye Movements in Natural Behaviour. Oxford,
New York: Oxford University Press.
Langer, J. (1987). The construction of meaning and the assessment of comprehension: An analysis of
reader performance on standardized test items. In R. O. Freedle & R. P. Duran (Eds.), Cognitive
and linguistic analyses of text performance (pp. 225-244). Norwood, NJ: Ablex.
Li, W. (1992). What is a test testing? An investigation of the agreement between students´ test taking
processes and test constructers´ presumptions. Unpublished MA thesis, Lancaster University,
Lancaster.
Messick, S. (1988). The Once and Future Issues of Validity: Assessing the Meaning and Consequences
of Measurement. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 33–46). Hillsdale, New
Jersey, Hove, London: Lawrence Earlbaum Associates.
Messick, S. (1995). Validity of Psychological Assessment: Validation of Inferences From Persons’
Responses and Performance as Scientific Inquiry Into Score Meaning. American Psychologist
50 (741–7499.
Mullis, I. V. S., Kennedy, A. M., Martin, M. O. & Sainsbury, M. (2006). PIRLS 2006 Assessment Framework
and Specifications. (2nd ed.). Chestnut Hill, MA: Boston College.
Ozuru, Y., Best, R., Bell, C., Witherspoon, A. & McNamara, D. S. (2007). Influence of Question Format and
Text Availability on the Assessment of Expository Text Comprehension. Cognition and
Instruction 25 (399-438).
Ozuru, Y., Dempsey, K., McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the
comprehension of science texts. Learning and Instruction 19 (228-242).
Paulson, E.J. & Henry, J. (2002). Does the Degrees of Reading Power Assessment Reflect the Reading
Process? An Eye-Movement Examination. Journal of Adolescent & Adult Literacy 46 (234–244).
Pearson, P. D. & Hamm, D. N. (2005). The Assessment of Reading Comprehension: A Review of
Practices – Past, Present and Future. In S. G. Paris & S. A. Stahl (Eds.), Children’s Reading
Comprehension and Assessment (pp. 13–69). Mahwah, New Jersey: Lawrence Erlbaum
Associates.
Raven, J.C. (1958). Standard Progressive Matrices: Sets A, B, C, D & E. Oxford, UK: Oxford Psychologists
Press Ltd.
Raven, J. C., Court, J. H. & Raven, J. (1988). Standard Progressive Matrices: 1988 Edition. London, UK:
Oxford Psychologists Press Ltd.
Rayner, K. (1992). Eye Movements and visual cognition: Scene perception during reading. New York:
Springer-Verlag.
Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research.
Psychological Bulletin 124 (372–422).
Rothkopf, E. Z. & Billington, M. J. (1979). Goal-guided learning from text: Inferring a descriptive
processing model from inspection times and eye movements. Journal of Educational
Psychology 71 (310–327).
Salmerón, L., Vidal-Abarca, E., Mana, A., Martínez, T., Gil, L. & Naumann, J. (submitted). Reading
Strategies in task oriented reading: The case of PISA-like tasks. Manuscript submitted for
publication.
Solheim, O. J. (2011). The Impact of Reading Self-Efficacy and Task Value on Reading Comprehension
Scores in Different Item Formats. Reading Psychology 32 (1-27).
Solheim, O. J. & Skaftun, A. (2009). The problem of semantic openness and constructed response.
Assessment in Education 16 (149-164).
Tai, R. H., Loehr, J. F. & Brigham, F. J. (2006). An exploration of the use of eye-gaze tracking to study
problem-solving on standardized science assessments. International Journal of Research &
Method in Education 29 (185–208).
Valencia, S. W. & Pearson, P. D. (1987). Reading Assessment: Time for a change. The Reading Teacher 40
(726–732).
Vidal-Abarca, E., Mãná, A. & Gil, L. (2010). Individual Differences for Self-Regulating Task-Oriented
Reading Activities. Journal of Educational Psychology 102 (817-826).
Downloads
Published
2017-08-24
How to Cite
Solheim, O. J., & Uppstad, P. H. (2017). Eye-tracking as a tool in process-oriented reading test validation. International Electronic Journal of Elementary Education, 4(1), 153–168. Retrieved from http://www.iejee.com/index.php/IEJEE/article/view/218
Issue
Section
Articles
