Development of a test series for learning progress assessment in reading using rule-based design

Development of a test series for learning progress assessment in reading using rule-based design

First Author: Natalie Förster -- University of Münster
Additional authors/chairs: 
Jörg-Tobias Kuhn; Jasmin Munske; Elmar Souvignier
Keywords: Progress Monitoring, Reading comprehension, Fluency, Item characterictics, Item Response Theory
Abstract / Summary: 

Purpose:
Aim of the present study was to investigate whether rule-based item design enables to develop equivalent reading tests that are useful for computer-based learning progress assessment.
Method:
We identified item properties from the literature that are supposed to affect the difficulty and efficiency of reading processes on word-, sentence-, and text level (e.g. number of orthographic neighbors, propositional density). Using a combination of four item properties for each item, we developed an item pool, in which four items followed the same construction rules, respectively, resulting in four design-equivalent tests with 46 items each.
To investigate whether these four tests were also psychometrically equivalent, N = 1930 second graders completed the computer-based tests. A partial sample of N = 306 students also finished standardized school tests for validation purposes. We estimated item and person parameters for accuracy and response time using bivariate IRT-models.
Results
All item properties significantly affected either item difficulty or response time but also interactions between properties were found. Moreover, the IRT-based test information functions indicated that the difficulty and time intensity of the four test forms was similar (all η2 ≤ .04). Finally, the correlations with standardized school tests indicate good convergent and discriminant validity of the new tests.
Conclusions
Overall, rule-based item design proved to be a promising way to develop equivalent reading tests. Item difficulties and response times could be predicted by item properties, especially on sentence and text level. As indicated by the interactions between item properties, however, item properties do not affect item difficulty and response times purely additive.