What Makes Reading Difficult? An Investigation of the Contribution of Passage, Task, and Reader Characteristics on Item Difficulty, Using Explanatory Item Response Models

What Makes Reading Difficult? An Investigation of the Contribution of Passage, Task, and Reader Characteristics on Item Difficulty, Using Explanatory Item Response Models

First Author: Yukie Toyama -- UC Berkeley
Keywords: Item Response Theory, Assessment, Comprehension difficulty, Text Complexity
Abstract / Summary: 

Purpose: Reading comprehension (RC) is a multi-faceted construct which manifests through complex interactions among the reader, the passage, and the task (RAND Reading Study Group, 2002). Drawing on the item difficulty modeling paradigm, this study examined how these three factors as well as their interactions affected comprehension difficulty.

Method: The study used explanatory item response models (De Boeck & Wilson, 2004) to analyze a vertically-scaled item response matrix from an operational online assessment. The dataset included a wide range of readers (n=10,547) as well as of passages (n=48), covering grades 1 through 12+. Two equivalent samples were created for model building and cross validation.

Results: Analyses indicated that it is text features, as measured by computational text analyzers (e.g., Lexile, Coh-Metrix), rather than task features, that explained over half the variance in item difficulty, after controlling for student general vocabulary knowledge. Specifically, sentence length, word frequency, syntactic simplicity, and temporality were consistently found to significantly affect RC difficulty. Further, small but significant interaction effects were found: in general, high vocabulary readers benefitted more from traditional textual affordances (e.g., shorter sentences, familiar words) than their peers with lower vocabulary knowledge, especially when questions asked them to recall specific localized information without accessing the source passage. However, reverse was true with temporality: passages with more time markers helped low vocabulary readers while it was low temporality passages that helped high vocabulary readers – the finding similar to ones reported in the literature (e.g., O’Reilly & McNamara, 2007).

Conclusions: A number of factors contribute to RC difficulties in complex ways. Ultimately understanding this complexity helps design targeted interventions that best facilitate students’ RC development.