Advances in cognitive diagnosis modeling

Resumen

Cognitive diagnosis models (CDMs) have shown a rapid development over the past decades. This set of restricted latent class models is the basis for a new psychometric framework where the dimensions underlying performance on the tests are assumed to be discrete. Notwithstanding the progress achieved, some aspects have not been fully explored. It is for this reason that this dissertation aims to contribute in three directions. (1) Broadening the area of application of CDMs. Empirical data is used to illustrate how CDMs provide a new approach that not only overcomes the limitations of the conventional methods for assessing the validity and reliability of situational judgment tests (SJTs) scores, but that also allows for a deeper understanding on what SJTs really measure. The data set comes from an application of a SJT that presents situations about student-related issues. (2) Evaluating item-level model fit statistics. Factors such as generating model, test length, sample size, item quality, and correlational structure are considered in two different Monte Carlo studies. The performance of several statistics and different strategies to cope with poor-quality data are discussed. Additionally, the two-step likelihood ratio test is introduced as a new index for item-level model comparison. (3) Introducing model comparison as a way of improving cognitive diagnosis computerized adaptive testing (CD-CAT) applications. Accuracy and item usage of a CD-CAT based on the combination of models selected with the new item-level model comparison statistic are explored under different calibration sample size, Q-matrix complexity, and item bank length conditions using Monte Carlo methods. The advantages of this approach over the application of a single reduced CDM or a general model are discussed. In general, the results of the studies included in this dissertation can be the basis for more reliable assessments and indicate the importance of selecting an appropriate psychometric framework. Item-level model selection emerges as a new and promising strategy to make the best of our data that can be generalized to other psychometric frameworks such as traditional item response theory.