Detection of Q-matrix misspecification using two criteria for validation of cognitive structures under the Least Squares Distance Model

  1. Sonia J. Romero 1
  2. Xavier G. Ordoñez 2
  3. Vicente Ponsoda 3
  4. Javier Revuelta 3
  1. 1 Universidad a Distancia de Madrid, Spain
  2. 2 Universidad Complutense de Madrid, Spain
  3. 3 Universidad Autónoma de Madrid, Spain
Show affiliations +
Journal:
Psicológica: Revista de metodología y psicología experimental

ISSN: 1576-8597

Year of publication: 2014

Volume: 35

Issue: 1

Pages: 149-169

Type: Article

DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Psicológica: Revista de metodología y psicología experimental

Abstract

Cognitive Diagnostic Models (CDMs) aim to provide information about the degree to which individuals have mastered specific attributes that underlie the success of these individuals on test items. The Q-matrix is a key element in the application of CDMs, because contains links item-attributes representing the cognitive structure proposed for solve the test. Using a simulation study we investigated the performance of two model-fit statistics (MAD and LSD) to detect misspecifications in the Q-matrix within the least squares distance modeling framework. The manipulated test design factors included the number of respondents (300, 500, 1000), attributes (1, 2, 3, 4), and type of model (conjunctive vs disjunctive). We investigated MAD and LSD behavior under correct Q-matrix specification, with Qmisspecifications and in a real data application. The results shows that the two model-fit indexes were sensitive to Q-misspecifications, consequently, cut points were proposed to use in applied context.

Bibliographic References

  • Corter, J. E. (1995). Using clustering methods to explore the structure of diagnostic tests. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 305-326). Hillsdale, NJ: Erlbaum.
  • de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343-362. doi:10.1111/j.1745-3984.2008.00069
  • de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130. doi: 10.3102/1076998607309474
  • de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333-353. doi: 10.1007/BF02295640
  • DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361-389). Hillsdale, NJ: Erlbaum.
  • Dimitrov, D. M. (2007). Least squares distance method of cognitive validation and analysis for binary items using their item response theory parameters. Applied Psychological
  • Measurement, 31, 367-387. doi: 10.1177/0146621606295199
  • Dimitrov, D. M., & Atanasov, D. V. (2011). Conjunctive and disjunctive extensions of the least squares distance model of cognitive diagnosis. Educational and Psychological Measurement. doi: 10.1177/0013164411402324
  • Dimitrov, D. M., & Raykov, T. (2003). Validation of cognitive structures: A structural equation modeling approach. Multivariate Behavioral Research, 38(1), 1-23. doi: 10.1207/S15327906MBR3801_1
  • Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49(2), 175-186. doi: 10.1007/BF02294171
  • Embretson, S. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125-150). Hillsdale, NJ: Erlbaum.
  • Fischer, G. H. (1995). The linear logistic test model. In G. H. Fisher & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 131-156). New York: Springer-Verlag.
  • Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277. doi: 10.1177/0146621604272623
  • Henson, R., Templin, J. & Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191-210. doi: 10.1007/s11336-008-9089-5
  • Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272. doi: 10.1177/01466210122032064
  • Lawson, C. L., & Hanson, R. J. (1974). Solving least squares problems. Englewood Cliffs, NJ: Prentice-Hall.
  • Medina-Diaz, M. (1993). Analysis of cognitive structure using the linear logistic test model and quadratic assignment. Applied Psychological Measurement, 17(2), 117-130. doi: 10.1177/014662169301700202
  • Rao, C. R. & Mitra, S. K. (1971). Generalized inverse of matrices and its applications. New York: Wiley.
  • Rizopoulos, D. (2006). ltm: An R package for latent variable modeling and item response analysis. Journal of Statistical Software, 17(5).
  • Romero, S. J. (2010). Propiedades y aplicaciones del método de las distancias mínimo cuadráticas (LSDM) para la validación y análisis de atributos cognitivos [Properties and applications of the least squares distance method (LSDM) for validation and analysis of cognitive attributes]. (Doctoral Dissertation in Spanish, Universidad Autónoma de Madrid). Retrieved at: http://digitooluam.greendata.es:1801/webclient/DeliveryManager?pid=30756
  • Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96. doi: 10.1177/0013164407301545
  • Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10(1), 55-73. doi: 10.2307/1164930
  • Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 327-359). illsdale, NJ: Erlbaum.
  • Templin, J. L. & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305. doi: 10.1037/1082-989X.11.3.287
  • von Davier, M. (2010). Hierarchical mixtures of diagnostic models. Psychological Test and Assessment Modeling, 52(1), 8-28.
Data source: Dialnet