Emotion Detection in Programming Learning: The Effects of Using Knowledge Estimates in Sensor-Free Models that Detect Student Confusion
DOI:
https://doi.org/10.5753/rbie.2024.3437Keywords:
Confusion detection, Sensor-free models, Machine learning, Student knowledge estimates, Programming learning, Emotional regulation in learning, Emotions in learningAbstract
Confusion is an emotion likely to occur in learning tasks involving complex content, such as in computer programming learning. When not regulated by the student, confusion can negatively affect learning. When regulated, it can lead to deeper levels of learning. The study described in this article sought to improve the performance of sensor-free models that detect student confusion while engaged in programming learning tasks. These models are interesting when integrated into programming tools because, by detecting student confusion during learning, the tool could intervene and assist the student in regulating his/her emotion. Related work trained confusion detection models using data from student interactions with the programming environment, such as data on keyboard and mouse movements. Our study hypothesized that incorporating data on student knowledge estimates into interaction data could improve the models' performance. We compared the performance of machine learning models trained with the hypothesis approach to models trained with the approach of related work. The models were trained with data collected from 62 students in programming classes over five months. The results presented positive evidence supporting our hypothesis. We also discussed scenarios where our approach is advantageous, such as the appropriate size of data segments, the best-performing algorithms, and the generalization power of the models for students of different educational levels.
Downloads
References
Agarwal, D., Baker, R., & Muraleedharan, A. (2020). Dynamic knowledge tracing through data driven recency weights. Em A. N. Rafferty, J. Whitehill, V. Cavalli-Sforza & C. Romero (Ed.), Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020) (pp. 725–729). [GS Search]
Alzoubi, O., D’Mello, S., & Calvo, R. A. (2012). Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals. IEEE Transactions on Affective Computing, 3(3), 298- 310. https://doi.org/10.1109/t-affc.2012.4. [GS Search]
Arguel, A., Lockyer, L., Kennedy, G., Lodge, J. M., & Pachman, M. (2019). Seeking optimal confusion: a review on epistemic emotion management in interactive digital learning environments. Interactive Learning Environments, 27(2), 200-210. https://doi.org/10.1080/10494820.2018.1457544. [GS Search]
Arroyo, I., Cooper, D., Burleson, W., Woolf, B., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. Frontiers in Artificial Intelligence and Applications, (1), 17-24. https://doi.org/10.3233/978-1-60750-028-5-17. [GS Search]
Ausubel, D. P., Novak, J. D., & Hanesian, H. (1978). Educational Psychology: A Cognitive View. Holt, Rinehart; Winston. [GS Search]
Badrinath, A., Wang, F., & Pardos, Z. (2021). pybkt: An accessible python library of bayesian knowledge tracing models. Proceedings of the 14th International Conference on Educational Data Mining. https://doi.org/10.48550/arXiv.2105.00385. [GS Search]
Baker, R. S. d., Corbett, A., Gowda, S. M., Wagner, A. Z., MacLaren, B. A., Kauffman, L. R., Mitchell, A. P., & Giguere, S. (2010). Contextual slip and prediction of student performance after use of an intelligent tutor. User Modeling, Adaptation, and Personalization: 18th International Conference, UMAP 2010, 52-63. https://doi.org/10.1007/978-3-642-13470-8_7. [GS Search]
Baker, R. S. d., Pardos, Z., Gowda, S. M., Nooraei, B. B., & Heffernan, N. T. (2011). Ensembling predictions of student knowledge within intelligent tutoring systems. User Modeling, Adaption and Personalization: 19th International Conference, UMAP 2011, 13-24. https://doi.org/10.1007/978-3-642-22362-4_2. [GS Search]
Beck, J. E., & Chang, K.-m. (2007). Identifiability: A fundamental problem of student modeling. Em C. Conati, K. McCoy & G. Paliouras (Ed.), International Conference on User Modeling (pp. 137–146). https://doi.org/10.1007/978-3-540-73078-1_17. [GS Search]
Bosch, N., Chen, Y., & D’Mello, S. (2014). It’s written on your face: Detecting affective states from facial expressions while learning computer programming. Intelligent Tutoring Systems, 39-44. https://doi.org/10.1007/978-3-319-07221-0_5. [GS Search]
Bosch, N., & D’Mello, S. (2017). The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education, 27(1), 181-206. https://doi.org/10.1007/s40593-015-0069-5. [GS Search]
Bosch, N., D’Mello, S., & Mills, C. (2013). What emotions do novices experience during their first computer programming learning session? https://doi.org/10.1007/978-3-642-39112-5_2. [GS Search]
Botelho, A. F., Baker, R. S., & Heffernan, N. T. (2017). Improving sensor-free affect detection using deep learning. Artificial Intelligence in Education, 40-51. https://doi.org/10.1007/978-3-319-61425-0_4. [GS Search]
Breiman, L. (2001). Random forests. Machine Learning, 45, 5-32. https://doi.org/10.1023/A:1010933404324. [GS Search]
Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing (TAC), 1(1), 18-37. https://doi.org/10.1109/T-AFFC.2010.1. [GS Search]
Chi, M. T., & Ohlsson, S. (2005). Complex Declarative Learning. Cambridge University Press. https://doi.org/10.1007/978-1-4419-1428-6_295. [GS Search]
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104. [GS Search]
Corbett, A., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4, 253-278. https://doi.org/10.1007/bf01099821. [GS Search]
Corbett, A., Kauffman, L., Maclaren, B., Wagner, A., & Jones, E. (2010). A Cognitive Tutor for genetics problem solving: Learning gains and student modeling. Journal of Educational Computing Research, 42(2), 219-239. https://doi.org/10.2190/EC.42.2.e. [GS Search]
Coto, M., Mora, S., Grass, B., & Murillo-Morera, J. (2021). Emotions and programming learning: systematic mapping. Computer Science Education, 32(1), 1-36. https://doi.org/10.1080/08993408.2021.1920816. [GS Search]
Craig, S. D., & et al. (2008a). Emote aloud during learning with AutoTutor: Applying the Facial Action Coding System to cognitive - Affective states during learning. Cognition and Emotion, 22(5), 777-788. https://doi.org/10.1080/02699930701516759. [GS Search]
de Oliveira Alves, M., Medeiros, F. P. A., & Melo, L. B. (2020). Levantamento do Estado da Arte sobre Aprendizagem baseada em Problemas na Educação a Distância e Híbrida. Anais do XXXI Simpósio Brasileiro de Informática na Educação, 61-71. https://doi.org/10.5753/cbie.sbie.2020.61. [GS Search]
Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7), 1895-1923. https://doi.org/10.1162/089976698300017197. [GS Search]
D’Mello, S. (2020). Big data in the science of learning. Em Big data in psychological research (pp. 203-225). American Psychological Association. https://doi.org/10.1037/0000193-010. [GS Search]
D’Mello, S., & Calvo, R. A. (2013). Beyond the basic emotions: what should affective computing compute? CHI'13 Extended Abstracts on Human Factors in Computing Systems, 2287-2294. https://doi.org/10.1145/2468356.2468751. [GS Search]
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145-157. https://doi.org/10.1016/j.learninstruc.2011.10.001. [GS Search]
D’Mello, S., & Graesser, A. C. (2014). Confusion. Em International Handbook of Emotions in Education (pp. 299-320). Routledge. https://doi.org/10.4324/9780203148211. [GS Search]
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153-170. https://doi.org/10.1016/j.learninstruc.2012.05.003. [GS Search]
D’Mello, S., Person, N., & Lehman, B. (2009). Antecedent-consequent relationships and cyclical patterns between affective states and problem solving outcomes. Frontiers in Artificial Intelligence and Applications, 200, 57-64. https://doi.org/10.3233/978-1-60750-028-5-57. [GS Search]
Felipe, D. A. M., Gutierrez, K. I. N., Quiros, E. C. M., & Vea, L. A. (2012). Towards the development of intelligent agent for novice c/c++ programmers through affective analysis of event logs. Proc. Int. MultiConference Eng. Comput. Sci, 1, 2012. [GS Search]
Fino, C. N. (2001). Vygotsky e a Zona de Desenvolvimento Proximal (ZDP): três implicações pedagógicas. Revista Portuguesa de educação, 14, 273-291. [GS Search]
Gong, Y., Beck, J. E., & Heffernan, N. T. (2010). Comparing knowledge tracing and performance factor analysis by using multiple model fitting procedures. Intelligent Tutoring Systems: 10th International Conference, ITS 2010, Pittsburgh, PA, USA, June 14-18, 2010, Procee- dings, Part I 10, 35–44. https://doi.org/10.1007/978-3-642-13388-6_8. [GS Search]
Gowda, S. M., Rowe, J. P., de Baker, R. S. J., Chi, M., & Koedinger, K. R. (2011). Improving Models of Slipping, Guessing, and Moment-By-Moment Learning with Estimates of Skill Difficulty. EDM, 2011, 199-208. [GS Search]
Graesser, A., Chipman, P., King, B., McDaniel, B., & D’Mello, S. (2007). Emotions and learning with autotutor. Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, 569-571. [GS Search]
Grafsgaard, J. F., Boyer, K. E., & Lester, J. C. (2011). Predicting facial indicators of confusion with hidden markov models. Affective Computing and Intelligent Interaction, 6974 LNCS(PART 1), 97-106. https://doi.org/10.1007/978-3-642-24600-5_13. [GS Search]
Halgren, E., & et al. (2002). N400-like Magnetoencephalography Responses Modulated by Semantic Context, Word Frequency, and Lexical Class in Sentences. NeuroImage, 17(3), 1101-1116. https://doi.org/10.1006/nimg.2002.1268. [GS Search]
Hess, U. (2003). Now you see it, now you don't-the confusing case of confusion as an emotion: Commentary on Rozin and Cohen (2003). Emotion, 3(1), 76-80. https://doi.org/10.1037/1528-3542.3.1.76. [GS Search]
Izard, C. E. (2010). The many meanings/aspects of emotion: Definitions, functions, activation, and regulation. Emotion Review, 2(4), 363-370. https://doi.org/10.1177/1754073910374661. [GS Search]
Jófili, Z. (2002). Piaget, Vygotsky, Freire e a construção do conhecimento na escola. Educação: teorias e práticas, 2(2), 191-208. [GS Search]
Kasurinen, J., & Nikula, U. (2009). Estimating programming knowledge with Bayesian knowledge tracing. ACM SIGCSE Bulletin, 41(3), 313-317. https://doi.org/10.1145/1595496.1562972. [GS Search]
Kautzmann, T. R., Ramos, G. d. O., & Jaques, P. A. (2022). O uso de estimativas de conhecimento do aluno em programação de computadores em modelos de detecção da emoção confusão livres de sensores. Anais do XXXIII Simpósio Brasileiro de Informática na Educação, 1196-1208. https://doi.org/10.5753/sbie.2022.225768. [GS Search]
Keltner, D., & Shiota, M. N. (2003). New displays and new emotions: A commentary on Rozin and Cohen (2003). Emotion, 3(1), 86-91. https://doi.org/10.1037/1528-3542.3.1.86. [GS Search]
Khajah, M., Lindsey, R. V., & Mozer, M. C. (2016). How deep is knowledge tracing? arXiv preprint arXiv:1604.02416. https://doi.org/10.48550/arXiv.1604.02416. [GS Search]
Knottnerus, J. A., & Tugwell, P. (2010). Real world research. Journal of clinical epidemiology, 63(10), 1051-1052. https://doi.org/10.1016/j.jclinepi.2010.08.001. [GS Search]
Koedinger, K. R., Corbett, A., & Perfetti, C. (2012). The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science, 36(5), 757-798. https://doi.org/10.1111/j.1551-6709.2012.01245.x. [GS Search]
Kubat, M. (2017). An introduction to machine learning (Vol. 2). Springer. https://doi.org/10.1007/978-3-319-63913-0. [GS Search]
Lee, D. M. C., Rodrigo, M. M. T., Baker, R. S., Sugay, J. O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. International Conference on Affective Computing and Intelligent Interaction, 175-184. https://doi.org/10.1007/978-3-642-24600-5_21. [GS Search]
Lehman, B., D’Mello, S., Strain, A., Mills, C., Gross, M., Dobbins, A., Wallace, P., Millis, K., & Graesser, A. (2013). Inducing and tracking confusion with contradictions during complex learning. International Journal of Artificial Intelligence in Education, 22, 85-105. https://doi.org/10.3233/JAI-130025. [GS Search]
Lin, C., & Chi, M. (2016). Intervention-bkt: incorporating instructional interventions into bayesian knowledge tracing. Intelligent Tutoring Systems: 13th International Conference, ITS 2016, Zagreb, Croatia, June 7-10, 2016. Proceedings 13, 208–218. https://doi.org/10.1007/978-3-319-39583-8_20. [GS Search]
MacDowell, K. A., & Mandler, G. (1989). Constructions of emotion: Discrepancy, arousal, and mood. Motivation and Emotion, 13(2), 105-124. https://doi.org/10.1007/bf00992957. [GS Search]
Mao, Y. (2018). Deep Learning vs. Bayesian Knowledge Tracing: Student Models for Interventions. Journal of educational data mining, 10(2). https://doi.org/10.5281/zenodo.3554691. [GS Search]
Mayer, J. D., Salovey, P., & Caruso, D. R. (2004). Emotional intelligence: Theory, findings, and implications. Psychological inquiry, 15(3), 197-215. [GS Search]
McDaniel, B., & et al. (2007). Facial Features for Affective State Detection in Learning Environments. Proceedings of the Annual Meeting of the Cognitive Science Society. [GS Search]
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153-157. https://doi.org/10.1007/bf02295996. [GS Search]
Molnar, C., Freiesleben, T., König, G., Herbinger, J., Reisinger, T., Casalicchio, G., Wright, M. N., & Bischl, B. (2023). Relating the partial dependence plot and permutation feature importance to the data generating process. World Conference on Explainable Artificial Intelligence, 456-479. https://doi.org/10.1007/978-3-031-44064-9_24. [GS Search]
Moon, T. K. (1996). The expectation-maximization algorithm. IEEE Signal processing magazine, 13(6), 47-60. https://doi.org/10.1109/79.543975. [GS Search]
Moors, A., Ellsworth, P. C., Scherer, K. R., & Frijda, N. H. (2013). Appraisal theories of emotion: State of the art and future development. Emotion Review, 5(2), 119-124. https://doi.org/10.1177/1754073912468165. [GS Search]
Nagatani, K., Zhang, Q., Sato, M., Chen, Y.-Y., Chen, F., & Ohkuma, T. (2019). Augmenting knowledge tracing by considering forgetting behavior. The world wide web conference, 3101-3107. https://doi.org/10.1145/3308558.3313565. [GS Search]
Pardos, Z., & Heffernan, N. (2010). Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm. Educational Data Mining 2010. [GS Search]
Pardos, Z., & Heffernan, N. T. (2011). KT-IDEM: Introducing item difficulty to the knowledge tracing model. User Modeling, Adaption and Personalization: 19th International Conference, UMAP 2011, Girona, Spain, July 11-15, 2011. Proceedings 19, 243-254. https://doi.org/10.1007/978-3-642-22362-4_21. [GS Search]
Pekrun, R. (2011). Emotions as drivers of learning and cognitive development. New perspectives on affect and learning technologies, 3, 23-39. https://doi.org/10.1007/978-1-4419-9625-1_3. [GS Search]
Pekrun, R., & Stephens, E. J. (2012). Academic emotions. Em APA Educational Psychology Handbook, Vol 2: Individual Differences and Cultural and Contextual Factors (pp. 3-31). American Psychological Association. https://doi.org/10.1037/13274-001. [GS Search]
Pekrun, R. (2014). Emotions and learning. [GS Search]
Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27, 313-350. https://doi.org/10.1007/s11257-017-9193-2. [GS Search]
Pelizzari, A., KriegL, M. d. L., Baron, M. P., Finck, N. T. L., & Dorocinski, S. I. (2002). Teoria da aprendizagem significativa segundo Ausubel. revista PEC, 2(1), 37-42. [GS Search]
Penmetsa, P. (2021). Investigate effectiveness of code features in knowledge tracing task on novice programming course. North Carolina State University. [GS Search]
Raposo, A. C., Maranhão, D., & Neto, C. S. (2019). Analise do modelo bkt na avaliacao da curva de aprendizagem de alunos de algoritmos. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), 30(1), 479. https://doi.org/10.5753/cbie.sbie.2019.479. [GS Search]
Raschka, S. (2018). Model evaluation, model selection, and algorithm selection in machine learning. https://doi.org/10.48550/arXiv.1811.12808. [GS Search]
Rodrigo, M. M. T., Baker, R. S. J., & Nabos, J. Q. (2010). The relationships between sequences of affective states and learner achievement. Proceedings of the 18th International Conference on Computers in Education, 56-60. [GS Search]
Rozin, P., & Cohen, A. B. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 3(1), 68-75. https://doi.org/10.1037/1528-3542.3.1.68. [GS Search]
Scherer, K. R. (2005). What are emotions? and how can they be measured? Social science information, 44(4), 695-729. https://doi.org/10.1177/0539018405058216. [GS Search]
Shwartz-Ziv, R., & Armon, A. (2022). Tabular data: Deep learning is not all you need. Information Fusion, 81, 84-90. https://doi.org/10.1016/j.inffus.2021.11.011. [GS Search]
Silvia, P. J. (2010). Confusion and interest: The role of knowledge emotions in aesthetic experience. Psychology of Aesthetics, Creativity, and the Arts, 4(2), 75-80. https://doi.org/10.1037/a0017081. [GS Search]
Slater, S., & Baker, R. S. (2018). Degree of error in Bayesian knowledge tracing estimates from differences in sample sizes. Behaviormetrika, 45(2), 475-493. https://doi.org/10.1007/s41237-018-0072-x. [GS Search]
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1988). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251-296. https://doi.org/10.1023/A:1022193728205. [GS Search]
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive Architecture and Instructional Design: 20 Years Later. Educational Psychology Review, 10, 251-296. https://doi.org/10.1007/s10648-019-09465-5. [GS Search]
Taksic, V. (2000). Convergent and divergent validity of the Emotional Skills and Competence Questionnaire. Comunicación presentada en XII Days of Psychology, Zadar, Croacia. [GS Search]
Tiam-Lee, T. J., & Sumi, K. (2018). Adaptive feedback based on student emotion in a system for programming practice. Intelligent Tutoring Systems, 243-255. https://doi.org/10.1007/978-3-319-91464-0_24. [GS Search]
Tiam-Lee, T. J., & Sumi, K. (2019). Analysis and prediction of student emotions while doing programming exercises. International Conference on Intelligent Tutoring Systems, 24-33. https://doi.org/10.1007/978-3-030-22244-4_4. [GS Search]
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. B. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 209-249. https://doi.org/10.1207/s1532690xci2103_01. [GS Search]
Vea, L., & Rodrigo, M. M. (2017). Modeling negative affect detector of novice programming students using keyboard dynamics and mouse behavior. Trends in Artificial Intelligence: PRICAI 2016 Workshops, 127-138. https://doi.org/10.1007/978-3-319-60675-0_11. [GS Search]
Wang, S., Han, Y., Wu, W., & Hu, Z. (2017). Modeling student learning outcomes in studying programming language course. 2017 Seventh International Conference on Information Science and Technology (ICIST), 263-270. https://doi.org/10.1109/icist.2017.7926768. [GS Search]
Wang, Y., Heffernan, N. T., & Heffernan, C. (2015). Towards better affect detectors: effect of missing skills, class features and common wrong answers. Proceedings of the fifth international conference on learning analytics and knowledge, 31-35. https://doi.org/10.1145/2723576.2723618. [GS Search]
Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2), 270-280. https://doi.org/10.1162/neco.1989.1.2.270. [GS Search]
Wong, T. T. (2015). Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition, 48, 2839-2846. https://doi.org/10.1016/j.patcog.2015.03.009. [GS Search]
Yang, T.-Y., Baker, R. S., Studer, C., Heffernan, N., & Lan, A. S. (2019). Active learning for student affect detection. Proceedings of the 12th International Conference on Educational Data Mining, 208-217. [GS Search]
Yudelson, M. V., Koedinger, K. R., & Gordon, G. J. (2013). Individualized bayesian knowledge tracing models. Artificial Intelligence in Education: 16th International Conference, AIED 2013, 171-180. https://doi.org/10.1007/978-3-642-39112-5_18. [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Tiago R. Kautzmann, Gabriel de O. Ramos, Patricia A. Jaques
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
