Evaluating Concurrent Validity and Classification Accuracy of a Short Web-based Adaptive Assessment Compared to Two Assessments Widely Used in the United States

Evaluating Concurrent Validity and Classification Accuracy of a Short Web-based Adaptive Assessment Compared to Two Assessments Widely Used in the United States

First Author: John Ferrara -- Taylor Associates Communications Inc.
Additional authors/chairs: 
Alexandra Spichtig; Jeffrey Pascoe; Kristin M. Gehsman
Keywords: Reading Assessment, Correlational studies, Assessment, Reading Ability
Abstract / Summary: 

Purpose:
Valid and reliable assessments of reading proficiency are essential to educators’ ability to make informed instructional decisions. This study evaluated the results of InSight, a web-based, adaptive reading assessment in relation to the Smarter Balanced (SBAC) and the Partnership for Assessment of Readiness for College and Careers (PARCC) English Language Arts (ELA) assessments currently used in eighteen states in the US. InSight can be administered in a group or individualized setting with minimal teacher oversight. Students typically need 30-40 minutes to complete the assessment and the results are immediately available. SBAC and PARCC ELA require more oversight and time to administer, usually between 3.5 and 4.5 hours, and results are provided within a few weeks to a few months after administration.

Method:
Concurrent validity and classification accuracy were analyzed using data from students in grades 3-8 who completed InSight and either the SBAC (n= 5,993) or PARCC (n=1,674) ELA assessments.

Results:
InSight showed strong concurrent validity (Pearson’s r >.70) and classification accuracy (lower bound of the Area Under the Curve (AUC) Confidence Interval > .80 and overall classification rate >75%) with the SBAC and PARCC ELA assessments across all grades measured.

Conclusions:
Reading assessments differ along dimensions such as accuracy, demands on teachers and students, impact on instructional time, and the timeliness of results. These findings suggest that InSight, a short web-based, adaptive reading assessment is a valid and efficient means of measuring students’ reading/ELA proficiency in grades 3-8. The ability for a short reading assessment to classify students’ reading/ELA proficiency accurately in comparison to widely-used assessments used for high-stakes accountability is critical for many instructional decisions such as tiered interventions for at-risk students.