The Programme for International Student Assessment (PISA) is the world's most extensive assessment of student's scholastic and general abilities, featuring assessments in over 70 countries. In the program's 2015 cycle, the assessment included a computerized measure of collaborative problem-solving skills (CPS) using virtual agents. Studies on the validity of this approach are sparse, however. To firstly investigate whether this computer-supported approach was validly assessing CPS, we obtained the original tasks from the Organization for Economic Co-operation and Development (OECD). We identified the extent to which the assessment is related to existing collaboration measures, including subjective self-rated collaboration, external teacher-rated collaboration, co-player collaboration, as well as reasoning and reading. The results of structural equation modeling and response surface analysis (N = 483) showed that self-rated collaboration, teacher-rated collaboration, co-player collaboration, and reasoning were moderately related to PISA 2015 CPS performance, even after controlling for individual differences in reading performance.