Predicting Problem-Solving Proficiency with Multiclass Hierarchical Classification Using Process Data

Volume
65
Year
2023
Issue number
1
Page range
145-278
Access date
06.12.2023
Relates to study/studies
PIAAC Cycle 1

Predicting Problem-Solving Proficiency with Multiclass Hierarchical Classification Using Process Data

A Machine Learning Approach

Abstract

Increased use of computer-based assessments has facilitated data collection processes that capture both response product data (i.e., correct and incorrect) and response process data (e.g., time-stamped action sequences). Evidence suggests a strong relationship between respondents' correct/incorrect responses and their problem-solving proficiency scores. However, few studies have reported the predictability of fine-grained process information on respondents' problem-solving proficiency levels and the degree of granularity needed for accurate prediction. This study uses process data from interactive problem-solving items in the Programme for the International Assessment of Adult Competencies (PIAAC) to predict proficiency levels with hierarchical classification methods. Specifically, we extracted aggregate-level process variables and item-specific sequences of problem-solving strategies. Two machine learning methods -- random forest and support vector machine -- affiliated with two multiclass hierarchical classification approaches (i.e., flat classification and hierarchical classification) were examined. Using seven problem-solving items from the U.S. PIAAC process data sample, we found that the hierarchical approach affiliated with any machine learning method performed moderately better than the flat approach in proficiency level prediction. This study demonstrates the feasibility of using process variables to classify respondents by problem-solving proficiency levels, and thus, supports the development of tailored instructions for adults at different levels.