Nonparametric Classification Method for Multiple-Choice Items in Cognitive Diagnostic Assessments
Abstract
Cognitive diagnostic models (CDMs) aim to estimate the mastery and nonmastery of attributes for examinees. Numerous CDMs have been developed; however most of them can only be used to analyze dichotomous responses. Multiple-choice (MC) items are always dichotomized to fit the data with these CDMs, which in turn, may cause information loss. In response to this issue, the MC-DINA model (de la Torre, 2009) was proposed and this model defined coded options as options requiring specific attributes. However, the MC-DINA model performs well with large-scale assessments but may become less effective when applied to small-scale data. In this study, a nonparametric classification approach for analyzing MC items (MC-NPC) is proposed. The MC-NPC method is based on the nonparametric classification (NPC; Chiu & Douglas, 2013) method, which classifies examinees to the group that has an ideal response vector closest to an observed item response vector by comparing it to all possible ideal item response vectors. The penalized Hamming distance was employed in the MC-NPC method to downplay the discrepancy between an ideal response of 0 and an observed response. Two simulation studies were conducted. The first demonstrated that the MC-NPC method outperforms the MC-DINA model and the DINA model and the NPC method for dichotomous data when the samples are small. The second study investigated the performance of these methods with misspecified Q-matrices and the results showed that the MC-NPC method remained superior to the NPC method with 5% of the misspecified noncoded options.