Measuring Real-time Student Engagement in Contextual Word Learning

Measuring Real-time Student Engagement in Contextual Word Learning

First Author: SungJin Nam -- University of Michigan
Additional authors/chairs: 
Kevyn Collins-Thompson; Leslie Hodges; Gwen Friskoff
Keywords: Training study, Vocabulary, Learning, Assessment, Attention
Abstract / Summary: 

This study reports a new measure of student engagement, based on real-time scoring of free response data from a contextual word learning (CWL) task.

Six participants inferred the meanings of novel words from single sentences that provided varying levels of support (hi, med, lo) and were normed using a "cloze" completion task. After participants viewed each target in context, they were asked to think of a near-synonym and type their answer into an intelligent tutoring system (ITS). There were 176 trials. Three of the participants were cued to "game" the task whenever possible for about half the trials by entering responses as quickly as possible. An online spell-check was used to discourage low-quality responses.

We used a measure of semantic similarity to grade the quality of responses on each trial. Cloze norms were used to flag responses that were 2+ standard deviations outside the normative range. Responses that were too fast (<300 ms) were also flagged. One confederate completed the task by typing the word "blank" during the gaming trials. Most of these errors were correctly flagged, but this also suggested the need for an additional measure to capture low variability across trials.

This measure will be used in a classroom study of CWL to flag repeated low-quality responses. Teachers can access a "dashboard" that displays results and flags students who are struggling or disengaged.