Korean Journal of Psychology : General

理쒓렐샇 寃깋

Korean Journal of Psychology : General - Vol. 40 , No. 4

[ Article ]
The Korean Journal of Psychology: General - Vol. 40, No. 4, pp. 539-566
ISSN: 1229-067X (Print)
Print publication date 25 Dec 2021
Received 10 Oct 2021 Accepted 25 Oct 2021
DOI: https://doi.org/10.22257/kjp.2021.12.40.4.539

컴퓨터 기반 적응적 심리 검사 제작을 위한 문항 선정 알고리즘으로서 Alternating Model Tree의 활용 가능성 탐색
윤정한 ; 이태헌
중앙대학교 심리학과

Investigating the Viability of Alternating Model Tree As An Item Selection Algorithm for Constructing Computerized Adaptive Psychological Testing
Jeong-Han Youn ; Taehun Lee
Department of Psychology, Chung-Ang University
Correspondence to : 이태헌, 중앙대학교 사회과학대학 심리학과 부교수, (155-756) 서울시 동작구 흑석로 84 Tel: 02-820-5124, E-mail: lee0267@cau.ac.kr

Funding Information ▼

초록

컴퓨터 기반 적응적 검사는 이전에 제시된 문항에 대한 반응을 기반으로 피검사자의 잠재 특질 수준을 추정하기에 가장 적절한 다음 문항을 선택해 출제함으로써 피검사자별 맞춤형 검사를 제시하는 컴퓨터 기반 검사형태다. 컴퓨터 기반 적응적 검사 제작의 핵심 요소 중 하나는 문항 선정 알고리즘이라 할 수 있으며, 최근 결정-트리를 이용한 적응적 심리 검사 구성에 대한 관심과 적용 사례가 늘어나고 있다. 결정-트리는 기계학습 분야에서 주로 연구되어온 예측 모형 중 하나로서 쉽게 해석 가능한 트리-구조를 가진다는 장점에도 불구하고 과적합 문제에 매우 취약하다는 것이 알려져 있다. 본 연구의 목적은 기계학습 분야에서 결정-트리의 대안으로 제시된 앙상블 모형 중에서 해석 가능한 트리-구조를 지닌 Alternating Model Tree (AMT)가 컴퓨터 기반 적응적 심리 검사 제작에 활용될 수 있는지 탐색하는 데 있다. 이를 위해 먼저 AMT의 작동 방식을 적응적 검사의 특징에 비추어 상술하였고, 검사 점수를 예측하는 AMT 기반의 적응적 검사와 결정-트리 기반의 적응적 검사의 예측 성능을 두 개의 심리 검사를 대상으로 비교하였다. 그 결과, AMT는 적응적 검사의 특징을 가지는 것으로 확인되었고, AMT 기반 적응적 심리 검사의 성능은 결정-트리 기반 적응적 심리 검사의 성능과 유사하거나 더 나은 결과를 보였다. 이러한 결과를 바탕으로 본 연구의 의의와 한계, 후속연구에 대한 제언 등을 논의하였다.

Abstract

Computerized adaptive testing (CAT) is a computer-administered test where the next question for estimating the examinee’s trait level is selected depending on his or her reponses to the previous items, resulting in tailored testing for each individual examinee. A defining feature of CAT stems from its item selection algorithms, among which both research interest and practical applications of decision-tree based CAT (DT-based CAT) have been rising recently. In the field of machine learning, however, it is well known that decision-trees, as a form of predictive models with simple and interpretable tree structures, can be vulnerable to the problem of overfitting or the problem of creating overly complex trees that do not generalize to newly observed data. Among various ensemble techniques developed to adequately address this problem, we the authors paid attention to the Alternating Model Tree (AMT) due to its interpretable tree-like structure. The purpose of this article is to investigate the viability of the Alternating Model Tree (AMT) as an item selection algorithm for constructing CAT. To this end, we first presented a detailed exposition of how AMT-based CAT can be constructed and then compared its performance with DT-based CAT using two sets of publicly available psychological test scores. The results provided supportive evidence that AMT-based CAT is viable, and that AMT-based CAT can predict test scores at least as accurate as DT-based CAT does. Based on our findings, we discuss implications, limitations, and directions of future studies.


Keywords: Computerized Adaptive Test, Decision Tree, Alternating Model Tree, Item Selection Algorithm
키워드: 적응적 심리 검사, 컴퓨터 기반 검사, 결정-트리, Alternating Model Tree, 문항 선정 알고리즘

Acknowledgments

이 논문은 2019년도 중앙대학교 CAU GRS 지원에 의하여 작성되었음.


References
1. Antal., M. (2013). On the use of elo rating for adaptive assessment. Studia Universitatis Babes-Bolyai, Informatica, 58(1), 29-41.
2. Babcock, B., & Weiss, D. J. (2009). Termination criteria in computerized adaptive tests: Variable-length CATs are not biased. In D. J. Weiss (Eds.), Proceedings of the 2009 GMAC conference on computerized adaptive testing (Vol. 14). http://www.psych.umn.edu/psylabs/CATCentral/
3. Baron-Cohen, S., Richler, J., Bisarya, D., Gurunathan, N., & Wheelwright, S. (2003). The systemizing quotient: an investigation of adults with Asperger syndrome or high-functioning autism, and normal sex differences. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 358(1430), 361-374. https://doi.org/10.1098/rstb.2002.1206
4. Baron-Cohen, S., & Wheelwright, S. (2004). The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. Journal of autism and developmental disorders, 34(2), 163-175. https://doi.org/10.1023/b:jadd.0000022607.19833.00
5. Bjorner, J. B., Chang, C. H., Thissen, D., & Reeve, B. B. (2007). Developing tailored instruments: item banking and computerized adaptive assessment. Quality of Life Research, 16(1), 95-108. https://doi.org/10.1007/s11136-007-9168-6 
6. Breiman, L., J. H. Friedman, R. A. Olshen, & Stone, C. J. (1984). Classification and Regression Trees. CRC Press.
7. Brownlee, J. (2016). Master Machine Learning Algorithms: discover how they work and implement them from scratch. Machine Learning Mastery.
8. Carlo, A. D., Barnett, B. S., & Cella, D. (2021). Computerized Adaptive Testing (CAT) and the Future of Measurement-Based Mental Health Care. Administration and Policy in Mental Health and Mental Health Services Research, 48, 729-731. https://doi.org/10.1007/s10488-021-01123-9
9. Delgado-Gomez, D., Baca-Garcia, E., Aguado, D., Courtet, P., & Lopez-Castroman, J. (2016). Computerized adaptive test vs. decision trees: development of a support decision system to identify suicidal behavior. Journal of affective disorders, 206, 204-209. https://doi.org/10.1016/j.jad.2016.07.032
10. Delgado-Gómez, D., Laria, J. C., & Ruiz-Hernández, D. (2019). Computerized adaptive test and decision trees: A unifying approach. Expert Systems with Applications, 117, 358-366. https://doi.org/10.1016/j.eswa.2018.09.052
11. Dietterich, T. G., & Kong, E. B. (1995). Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Department of Computer Science, Oregon State University.
12. Domingos, P. (2000). A unified bias-variance decomposition. In P. Langley (Eds.), Proceedings of 17th International Conference on Machine Learning (pp. 231-238). Morgan Kaufmann.
13. Frank, E., Mayo, M., & Kramer, S. (2015). Alternating model trees. In R. L. WainWright (Eds.), Proceedings of the 30th annual ACM symposium on applied computing (pp. 871-878). ACM. https://doi.org/10.1145/2695664.2695848
14. Gibbons, R. D., Hooker, G., Finkelman, M. D., Weiss, D. J., Pilkonis, P. A., Frank, E., Moore, T. & Kupfer, D. J. (2013). The computerized adaptive diagnostic test for major depressive disorder (CAD-MDD): a screening tool for depression. The Journal of clinical psychiatry, 74(7), 669-674. https://doi.org/10.4088/jcp.12m08338 
15. Gibbons, R. D., Weiss, D. J., Frank, E., & Kupfer, D. (2016). Computerized adaptive diagnosis and testing of mental health disorders. Annual review of clinical psychology, 12, 83-104. https://doi.org/10.1146/annurev-clinpsy-021815-093634
16. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning. Springer. https://doi.org/10.1007/b94608
17. Hsueh, I. P., Chen, J. H., Wang, C. H., Chen, C. T., Sheu, C. F., Wang, W. C., Hou, W. H., & Hsieh, C. L. (2010). Development of a computerized adaptive test for assessing balance function in patients with stroke. Physical therapy, 90(9), 1336-1344. https://doi.org/10.2522/ptj.20090395
18. Hu, Y., Cai, Y., Tu, D., Guo, Y., & Liu, S. (2020). Development of a Computerized Adaptive Test for Separation Anxiety Disorder Among Adolescents. Frontiers in Psychology, 11, 1077. https://doi.org/10.3389/fpsyg.2020.01077
19. Istiyono, E., Dwandaru, W. S. B., Setiawan, R., & Megawati, I. (2020). Developing of Computerized Adaptive Testing to Measure Physics Higher Order Thinking Skills of Senior High School Students and Its Feasibility of Use. European Journal of Educational Research, 9(1), 91-101. https://doi.org/10.12973/eu-jer.9.1.91
20. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. Springer. https://doi.org/10.1007/978-1-4614-7138-7
21. Kimura, T. (2017). The impacts of computer adaptive testing from a variety of perspectives. Journal of educational evaluation for health professions, 14(12),1-5. https://doi.org/10.3352/jeehp.2017.14.12
22. Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Springer. https://doi.org/10.1007/978-1-4614-6849-3
23. Lai, M. C., Lombardo, M. V., Chakrabarti, B., Ecker, C., Sadek, S. A., Wheelwright, S. J., Murphy, D. G. M., Suckling, J., Bullmore, E. T., MRC AIMS Consortium & Baron-Cohen, S. (2012). Individual differences in brain structure underpin empathizing-systemizing cognitive styles in male adults. Neuroimage, 61(4), 1347-1354. https://doi.org/10.1016/j.neuroimage.2012.03.018
24. Linden, W. J., van der Linden, W. J., & Glas, C. A. (Eds.). (2000). Computerized adaptive testing: Theory and practice. Springer. https://doi.org/10.1007/0-306-47531-6
25. Ma, X. (2018). Using classification and regression trees: A practical primer. IAP.
26. Magis, D., Yan, D., & Von Davier, A. A.(2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer. https://doi.org/10.1007/978-3-319-69218-0
27. Maimon, O. Z., & Rokach, L. (2014). Data mining with decision trees: theory and applications (Vol.81). World scientific.
28. Michel, P., Baumstarck, K., Loundou, A., Ghattas, B., Auquier, P., & Boyer, L. (2018). Computerized adaptive testing with decision regression trees: an alternative to item response theory for quality of life measurement in multiple sclerosis. Patient preference and adherence, 12, 1043. https://doi.org/10.2147/ppa.s162206
29. Muraki, E. (1990). Fitting a polytomous item response model to Likert-type data. Applied Psychological Measurement, 14(1), 59-71. https://doi.org/10.1177/014662169001400106
30. Myles, A. J., Feudale, R. N., Liu, Y., Woody, N. A., & Brown, S. D. (2004). An introduction to decision tree modeling. Journal of Chemometrics, 18(6), 275-285. https://doi.org/10.1002/cem.873
31. Peute, L., Scheeve, T., & Jaspers, M. (2020). Classification and Regression Tree and Computer Adaptive Testing in Cardiac Rehabilitation: Instrument Validation Study. Journal of medical Internet research, 22(1), e12509. https://doi.org/10.2196/preprints.12509
32. Raskin, R. N., & Hall, C. S. (1979). A narcissistic personality inventory. Psychological Reports, 45, 590.
33. Raskin, R., & Terry, H. (1988). A principal-components analysis of the Narcissistic Personality Inventory and further evidence of its construct validity. Journal of personality and social psychology, 54(5), 890. https://doi.org/10.1037/0022-3514.54.5.890
34. Rezaie, M., & Golshan, M. (2015). Computer adaptive test (CAT): Advantages and limitations. International Journal of Educational Investigations, 2(5), 128-137.
35. Seo, D. G., Lee, S. M., Kim, J. N., Choi, S. W., Chae, J. M., Jung, S. H., Cho, S. K., Kim, M. K. & Ebesutani, C. (2019). Psychometric Methods and Validation of Short Form for the Psychological Scale: Based on the Korean dysfunctional depression scale. Korean Journal of Psychology: General 38(1), 75-102. http://dx.doi.org/10.22257/kjp.2019.3.38.1.75
36. Šerbec, I. N., Žerovnik, A., & Rugelj, J.(2011). Adaptive assessment based on decision trees and decision rules. In A. Verbraeck, M. Helfert, J. Cordeiro, & B. Shishkov (Eds.), CSEDU 2011-Proceedings of the 3rd International Conference on Computer Supported Education (pp. 473-479). SciTePress. https://doi.org/10.5220/0003521104730479
37. Song, Y. Y., & Ying, L. U. (2015). Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry, 27(2), 130. https://doi.org/10.11919/j.issn.1002-0829.215044
38. Sug, H. (2009). An effective sampling method for decision trees considering comprehensibility and accuracy. WSEAS Transactions on Computers, 8(4), 631-640.
39. Thissen, D. E., & Wainer, H. E. (2001). Test scoring. Lawrence Erlbaum Associates Publishers. https://doi.org/10.4324/9781410604729
40. Twenge, J. M., Konrath, S., Foster, J. D., Keith Campbell, W., & Bushman, B. J. (2008). Egos inflating over time: A cross‐temporal meta‐analysis of the Narcissistic Personality Inventory. Journal of personality, 76(4), 875-902. https://doi.org/10.1111/j.1467-6494.2008.00507.x
41. Ueno, M., & Songmuang, P. (2010). Computerized adaptive testing based on decision tree. In B. Werner (Eds.), 2010 10th IEEE International Conference on Advanced Learning Technologies (pp. 191-193). IEEE Computer Society Press. https://doi.org/10.1109/icalt.2010.58
42. Van der Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of adaptive testing (pp. 3-30). Springer. https://doi.org/10.1007/978-0-387-85461-8_1
43. van der Oest, M. J., Porsius, J. T., MacDermid, J. C., Slijper, H. P., & Selles, R. W. (2020). Item reduction of the patient-rated wrist evaluation using decision tree modelling. Disability and rehabilitation, 42(19), 2758-2765. https://doi.org/10.1080/09638288.2019.1566407
44. Walter, O. B., Becker, J., Bjorner, J. B., Fliege, H., Klapp, B. F., & Rose, M. (2007). Development and evaluation of a computer adaptive test for ‘Anxiety’(Anxiety-CAT). Quality of Life Research, 16(1), 143-155. https://doi.org/10.1007/s11136-007-9191-7
45. Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., & Mislevy, R. J. (2000). Computerized adaptive testing: A primer. Routledge. https://doi.org/10.4324/9781410605931
46. Winarno, D., & Si, S. (2018). Computerized Adaptive Testing (CAT) Using Triangle Decision Tree Method. International Journal of Science and Research, 7(5), 552-560. https://doi.org/10.21275/ART20182213
47. Wu, C. C., Chen, Y. L., Liu, Y. H., & Yang, X. Y. (2016). Decision tree induction with a constrained number of leaf nodes. Applied Intelligence, 45(3), 673-685. https://doi.org/10.1007/s10489-016-0785-z
48. Yan, D., Lewis, C., & Stocking, M. (2004). Adaptive testing with regression trees in the presence of multidimensionality. Journal of Educational and Behavioral Statistics, 29(3), 293-316. https://doi.org/10.3102/10769986029003293
49. Yan, D., Von Davier, A. A., & Lewis, C. (Eds.). (2016). Computerized multistage testing: Theory and applications. CRC Press.