Sensor-free Affect Detection in Learning Environments: A Systematic Literature Review
DOI:
https://doi.org/10.5753/rbie.2024.4362Keywords:
Sensor-Free Affect Detection , Systematic literature review, Emotional Learning Environments, Emotion Detection, Computer-Based Learning EnvironmentsAbstract
Emotions and affective states influence cognition and learning processes. Computer-based learning environments (CBLEs) capable of detecting and adapting to these states significantly enhance learning outcomes. However, practical constraints often hinder the deployment of sensor-based affect detection in CBLEs, especially for large-scale or long-term use. Consequently, sensor-free affect detection, reliant solely on interaction logs, emerges as a promising alternative. This paper offers a comprehensive literature review on sensor-free affect detection, covering frequently identified affective states, methodologies for sensor development, CBLE attributes, and research trends. Despite the field's maturity, there's ample room for further exploration. Future research should focus on improving sensor-free detection models, collecting more samples of underrepresented emotions, and refining model development practices. Additionally, efforts should be made to integrate models into CBLEs for real-time detection, provide meaningful interventions based on detected emotions, and deepen understanding of emotions' impact on learning. Key suggestions include comparing data collection techniques, optimizing duration granularity, establishing shared databases, and ensuring model source code accessibility.
Downloads
References
Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2004). Toward tutoring help seeking. International Conference on Intelligent Tutoring Systems (ITS), 227–239. https://doi.org/10.1007/978-3-540-30139-4_22. [GS Search].
Ali, A., Shamsuddin, S. M., & Ralescu, A. L. (2013). Classification with class imbalance problem. Int. J. Advance Soft Compu. Appl, 5(3), 176–204. [GS Search].
Andres, J. M. A. L., Ocumpaugh, J., Baker, R., Slater, S., Paquette, L., Jiang, Y., Karumbaiah, S., Bosch, N., Munshi, A., Moore, A., & Biswas, G. (2019). Affect sequences and learning in betty’s brain. International Conference on Learning Analytics and Knowledge (LAK), 383–390. https://doi.org/10.1145/3303772.3303807. [GS Search].
Arroyo, I., & et al. (2009). Emotion sensors go to school. International Conference on Artificial Intelligence in Education (AIED), 200, 17–24. https://doi.org/10.3233/978-1-60750-028-5-17. [GS Search].
Arroyo, I., Muldner, K., Schultz, S., Burleson, W., Wixon, N., & Woolf, B. P. (2016). Addressing affective states with empathy and growth mindset. 24th ACM Conference on User Modeling, Adaptation and Personalisation (UMAP), 314–319. https://doi.org/10.1007/978-3-319-39583-8_35. [GS Search].
Azevedo, R., & Aleven, V. (2013). International handbook of metacognition and learning technologies (Vol. 26). Springer. [GS Search].
Baker, R., Corbett, A., Koedinger, K., & Wagner, A. (2004). Off-task behavior in the cognitive tutor classroom: When students game the system. ACM CHI 2004: Computer-Human Interaction, 383–390. https://doi.org/10.1145/985692.985741. [GS Search].
Baker, R. (2007). Modeling and understanding students' off-task behavior in intelligent tutoring systems. Proceedings of the SIGCHI conference on Human factors in computing systems (ACM CHI), 1059–1068. https://doi.org/10.1145/1240624.1240785. [GS Search].
Baker, R., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003. [GS Search].
Baker, R., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Salvi, A., Aleven, V., Kusbit, G. W., Ocumpaugh, J., & Rossi, L. (2012). Towards sensor-free affect detection in cognitive tutor algebra: Population validity for educational data mining. British Journal of Educational Technology, 45(3), 487–501. https://doi.org/10.1111/bjet.12156. [GS Search].
Baker, R., & Ocumpaugh, J. (2014). Interaction-based affect detection in educational software. In R. A. Calvo, S. K. D’Mello, J. Gratch, & A. Kappas (Eds.), The oxford handbook of affective computing. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199942237.013.009. [GS Search].
Baker, R., Ocumpaugh, J., Gowda, S. M., Kamarainen, A. M., & Metcalf, S. J. (2014). Extending log-based affect detection to a multi-user virtual environment for science. International Conference on User Modeling, Adaptation, and Personalization, 290–300. https://doi.org/10.1007/978-3-319-08786-3_25. [GS Search].
Bloom, B. S. (1977). Human characteristics and school learning. (Vol. 10). McGraw-Hill. https://doi.org/10.2307/1478496. [GS Search].
Bosch, N., Chen, H., D’Mello, S., Baker, R., & Shute, V. (2015). Accuracy vs. availability heuristic in multimodal affect detection in the wild. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 267–274. https://doi.org/10.1145/2818346.2820739. [GS Search].
Bosch, N., & D’Mello, S. (2017). The affective experience of novice computer programmers. IJAIED, 27(1), 181–206. https://doi.org/10.1007/978-3-030-23204-7_25. [GS Search].
Bosch, N., D’Mello, S., & Mills, C. (2013). What emotions do novices experience during their first computer programming learning session? International Conference on Artificial Intelligence in Education (AIED), 11–20. https://doi.org/10.1007/978-3-642-39112-5_2. [GS Search].
Botelho, A. F., Baker, R., & Heffernan, N. T. (2017). Improving sensor-free affect detection using deep learning. International Conference on Artificial Intelligence in Education (AIED), 40–51. https://doi.org/10.1007/978-3-319-61425-0_4. [GS Search].
Botelho, A. F., Baker, R., & Heffernan, N. T. (2019). Machine-learned or expert-engineered features? exploring feature engineering methods in detectors of student behavior and affect. International Conference on Educational Data Mining (ICEDM). [GS Search].
Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37. https://doi.org/10.1109/t-affc.2010.1. [GS Search].
Carroll, J. B. (1963). A model of school learning. Teachers college record, 3(8), 155–167. https://doi.org/10.1177/016146816306400801. [GS Search].
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). Smote: Synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321–357. https://doi.org/10.1613/jair.953. [GS Search].
Cocea, M., Hershkovitz, A., & Baker, R. (2009). The impact of off-task and gaming behaviors on learning: Immediate or aggregate? International Conference on Artificial Intelligence in Education (AIED), 507–514. https://doi.org/10.3233/978-1-60750-028-5-507. [GS Search].
Cocea, M., & Weibelzahl, S. (2010). Disengagement detection in online learning: Validation studies and perspectives. IEEE transactions on learning technologies, 4(2), 114–124. https://doi.org/10.1109/tlt.2010.14. [GS Search].
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with autotutor. Journal of Eeducational Media, 29(3), 241–250. https://doi.org/10.1080/1358165042000283101. [GS Search].
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal performance. Harper; Row. [GS Search].
DeFalco, J. A., Rowe, J. P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., Mott, B. W., Baker, R., & Lester, J. C. (2018). Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education (IJAIED), 28(2), 152–193. https://doi.org/10.1007/s40593-017-0152-1. [GS Search].
de Morais, F., & Jaques, P. A. (2023). The dynamics of brazilian students’ emotions in digital learning systems: Investigating the interaction between gender and emotions duration. International Journal of Artificial Intelligence in Education (IJAIED), ((in press)). https://doi.org/10.1007/s40593-023-00339-0. [GS Search].
de Morais, F., Kautzmann, T. R., Bittencourt, I. I., & Jaques, P. A. (2019). EmAP-ML: A protocol of emotions and behaviors annotation for machine learning labels. European Conference for Technology-Enhanced Learning (EC-TEL). https://doi.org/10.1007/978-3-030-29736-7_37. [GS Search].
D’Mello, S. (2011). Dynamical emotions: Bodily dynamics of affect during problem solving. Proceedings of the Annual Meeting of the Cognitive Science Society, 33(33). [GS Search].
D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082. https://doi.org/10.1037/a0032674. [GS Search].
D’Mello, S., & Calvo, R. A. (2013). Beyond the basic emotions: What should affective computing compute? CHI'13 Extended Abstracts on Human Factors in Computing Systems, 2287–2294. https://doi.org/10.1145/2468356.2468751. [GS Search].
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001. [GS Search].
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. https://doi.org/10.1016/j.learninstruc.2012.05.003. [GS Search].
D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., & Graesser, A. (2010). A time for emoting: When affect-sensitivity is and isn't effective at promoting deep learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (pp. 245–254, Vol. 6094 LNCS). https://doi.org/10.1007/978-3-642-13388-6_29. [GS Search].
D’Mello, S. K., Craig, S. D., Sullins, J., & Graesser, A. (2006). Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue. International Journal of Artificial Intelligence in Education (IJAIED), 3–28. [GS Search].
Fredrickson, B. L. (1998). What good are positive emotions? Review of General Psychology, 2(3), 300–319. https://doi.org/10.1037/1089-2680.2.3.300. [GS Search].
Frenzel, A. C., Pekrun, R., & Goetz, T. (2007). Girls and mathematics—a “hopeless” issue? a control-value approach to gender differences in emotions towards mathematics. European Journal of Psychology of Education, 22(4), 497–514. https://doi.org/10.1007/BF03173468. [GS Search].
Graesser, A., & D’Mello, S. (2012). Emotions during the learning of difficult material. In Psychology of learning and motivation (pp. 183–225, Vol. 57). Elsevier. https://doi.org/10.1016/b978-0-12-394293-7.00005-4. [GS Search].
Graesser, A., D’Mello, S., & Strain, A. C. (2014). Emotions in advanced learning technologies. In International handbook of emotions in education (pp. 483–503). Routledge. [GS Search].
Hembree, R. (1988). Correlates, causes, effects, and treatment of test anxiety. Review of Educational Research, 58(1), 47–77. https://doi.org/10.3102/00346543058001047. [GS Search].
Henderson, N., Rowe, J., Paquette, L., Baker, R., & Lester, J. (2020). Improving affect detection in game-based learning with multimodal data fusion. International Conference on Artificial Intelligence in Education (AIED), 228–239. https://doi.org/10.1007/978-3-030-52237-7_19. [GS Search].
Hidi, S. (2006). Interest: A unique motivational variable. Educational Research Review, 1(2), 69–82. https://doi.org/10.1016/j.edurev.2006.09.001. [GS Search].
Hutt, S., Grafsgaard, J. F., & D’Mello, S. (2019). Time to scale: Generalizable affect detection for tens of thousands of students across an entire school year. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/3290605.3300726. [GS Search].
Hyde, J. S., Fennema, E., Ryan, M., Frost, L. A., & Hopp, C. (1990). Gender comparisons of mathematics attitudes and affect: A meta-analysis. Psychology of Women Quarterly, 14(3), 299–324. https://doi.org/10.1111/j.1471-6402.1990.tb00022.x. [GS Search].
Jensen, E., Hutt, S., & D’Mello, S. K. (2019). Generalizability of sensor-free affect detection models in a longitudinal dataset of tens of thousands of students. International Educational Data Mining Society. [GS Search].
Jiang, Y., Bosch, N., Baker, R., Paquette, L., Ocumpaugh, J., Andres, J. M. A. L., Moore, A. L., & Biswas, G. (2018). Expert feature-engineering vs. deep neural networks: Which is better for sensor-free affect detection? AIED, 198–211. https://doi.org/10.1007/978-3-319-93843-1_15. [GS Search].
Kai, S., Paquette, L., Baker, R., Bosch, N., D’Mello, S., Ocumpaugh, J., Shute, V., & Ventura, M. (2015). A comparison of video-based and interaction-based affect detectors in physics playground. International Conference on Educational Data Mining (ICEDM), 44–53. https://doi.org/10.1007/978-3-319-19773-9_5. [GS Search].
Kort, B., Reilly, R., & Picard, R. W. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion. Proceedings IEEE international conference on advanced learning technologies, 43–46. https://doi.org/10.1109/ICALT.2001.943850. [GS Search].
Lee, D. M. C., Rodrigo, M. M. T., d Baker, R., Sugay, J. O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. International Conference on Affective Computing and Intelligent Interaction, 175–184. https://doi.org/10.1007/978-3-642-24600-5_21. [GS Search].
Lee, S. W., Kelly, K. E., & Nyre, J. E. (1999). Preliminary report on the relation of students’ on-task behavior with completion of school work. Psychological Reports, 84(1), 267–272. https://doi.org/10.2466/pr0.1999.84.1.267. [GS Search].
Lehman, B., D’Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education, 15(3), 184–194. https://doi.org/10.1016/j.iheduc.2012.01.002. [GS Search].
Lehman, B., D’Mello, S., & Graesser, A. (2012). Interventions to regulate confusion during learning. International Conference on Intelligent Tutoring Systems (ITS), 576–578. https://doi.org/10.1007/978-3-642-30950-2_75. [GS Search].
Lehman, B., & et al. (2008). What are you feeling? investigating student affective states during expert human tutoring sessions. ITS, 50–59. https://doi.org/10.1007/978-3-540-69132-7_10. [GS Search].
Litman, D., & Forbes-Riley, K. (2014). Evaluating a spoken dialogue system that detects and adapts to user affective states. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 181–185. [GS Search].
Miserandino, M. (1996). Children who do well in school: Individual differences in perceived competence and autonomy in above-average children. Journal of educational psychology, 88(2), 203. https://doi.org/10.1037/0022-0663.88.2.203. [GS Search].
Mitchell, T. M., & Mitchell, T. M. (1997). Machine learning (Vol. 1). McGraw-hill. [GS Search].
Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487–501. https://doi.org/10.1111/bjet.12156. [GS Search].
Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T. (2015). Baker rodrigo ocumpaugh monitoring protocol (bromp) 2.0 technical and training manual. New York, NY and Manila, Philippines: Teachers College, Columbia University and Ateneo Laboratory for the Learning Sciences. [GS Search].
Paquette, L., Baker, R., Sao Pedro, M. A., Gobert, J. D., Rossi, L., Nakama, A., & KauffmanRogoff, Z. (2014). Sensor-free affect detection for a simulation-based science inquiry learning environment. ITS, 1–10. https://doi.org/10.1007/978-3-319-07221-0_1. [GS Search].
Paquette, L., Rowe, J., Baker, R., Mott, B., Lester, J., DeFalco, J., Brawner, K., Sottilare, R., & Georgoulas, V. (2016). Sensor-free or sensor-full: A comparison of data modalities in multi-channel affect detection. International Conference on Educational Data Mining (EDM). [GS Search].
Pardos, Z. A., Baker, R., San Pedro, M. O., Gowda, S. M., & Gowda, S. M. (2014). Affective states and state tests: Investigating how affect and engagement during the school year predict end-of-year learning outcomes. JLA, 1(1), 107–128. https://doi.org/10.18608/jla.2014.11.6. [GS Search].
Patrick, B. C., Skinner, E. A., & Connell, J. P. (1993). What motivates children’s behavior and emotion? joint effects of perceived control and autonomy in the academic domain. Journal of Personality and social Psychology, 65(4), 781. https://doi.org/10.1037/0022-3514.65.4.781. [GS Search].
Pedro, M. O., Baker, R., Bowers, A., & Heffernan, N. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. Educational Data Mining 2013. [GS Search].
Pekrun, R. (2007). Emotions in students' scholastic development. In The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 553–610). Springer. https://doi.org/10.1007/1-4020-5742-3_13. [GS Search].
Pekrun, R., et al. (2014). Emotions and learning (Vol. 24). International Academy of Education (IAE) Geneva, Switzerland. [GS Search].
Pekrun, R. (2016). Academic emotions. Handbook of motivation at school, 2, 120–144. https://doi.org/10.4324/9781138609877-ree210-1. [GS Search].
Pekrun, R., Elliot, A. J., & Maier, M. A. (2006). Achievement goals and discrete achievement emotions: A theoretical model and prospective test. Journal of educational Psychology, 98(3), 583. https://doi.org/10.1037/0022-0663.98.3.583. [GS Search].
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002a). Positive emotions in education. In Beyond coping: Meeting goals, visions, and challenges (pp. 149–173). Oxford University Press. https://doi.org/10.1093/med:psych/9780198508144.003.0008. [GS Search].
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002b). Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational psychologist, 37(2), 91–105. https://doi.org/10.4324/9781410608628-4. [GS Search].
Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In Handbook of research on student engagement (pp. 259–282). Springer. https://doi.org/10.1007/978-3-031-07853-8_6. [GS Search].
Perkins, H. V. (1965). Classroom behavior and underachievement. American Educational Research Journal, 2(1), 1–12. https://doi.org/10.3102/00028312002001001. [GS Search].
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008). Systematic mapping studies in software engineering. EASE, 8, 68–77. https://doi.org/10.14236/ewic/ease2008.8. [GS Search].
Porayska-Pomsta, K., Mavrikis, M., D'Mello, S., & et al. (2013). Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education (IJAIED), 22(3), 107–140. https://doi.org/10.1007/s40593-023-00346-1. [GS Search].
Reis, H., Alvares, D., Jaques, P., & Isotani, S. (2018). Analysis of permanence time in emotional states: A case study using educational software. Intelligent Tutoring Systems, 180–190. https://doi.org/10.1007/978-3-319-91464-0_18. [GS Search].
Rodrigo, M. M. T., Baker, R., Jadud, M. C., Amarra, A. C. M., Dy, T., Espejo-Lahoz, M. B. V., Lim, S. A. L., Pascua, S. A., Sugay, J. O., & Tabanao, E. S. (2009). Affective and behavioral predictors of novice programmer achievement. Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education, 156–160. https://doi.org/10.1145/1595496.1562929. [GS Search].
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172. https://doi.org/10.1037/0033-295x.110.1.145. [GS Search].
Sabourin, J., Mott, B., & Lester, J. C. (2011). Modeling learner affect with theoretically grounded dynamic bayesian networks. International Conference on Affective Computing and Intelligent Interaction, 286–295. https://doi.org/10.1007/978-3-642-24600-5_32. [GS Search].
Salmeron-Majadas, S., Santos, O. C., & Boticario, J. G. (2014). An evaluation of mouse and keyboard interaction indicators towards non-intrusive and low cost affective modeling in an educational context. Procedia Computer Science. [GS Search].
Scherer, K. R. (2000). Psychological models of emotion. The neuropsychology of emotion, 137(3), 137–162. [GS Search].
Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729. https://doi.org/10.1177/0539018405058216. [GS Search].
Smeets, R., Broaekman, F., & Bouwers, E. (2019). Affect detection in home-based educational software for young children. International Journal of Educational Research, 107, 101746. https://doi.org/10.1016/j.ijer.2021.101746. [GS Search].
Tiam-Lee, T. J., & Sumi, K. (2019). Analysis and prediction of student emotions while doing programming exercises. In International conference on intelligent tutoring systems (its) (pp. 24–33). Springer. https://doi.org/10.1007/978-3-030-22244-4_4. [GS Search].
Vea, L., Rodrigo, M., et al. (2016). Modeling negative affect detector of novice programming students using keyboard dynamics and mouse behavior. Pacific Rim International Conference on Artificial Intelligence, 127–138. https://doi.org/10.1007/978-3-319-60675-0_11. [GS Search].
Verduyn, P., Delaveau, P., Rotgé, J.-Y., Fossati, P., & Mechelen, I. V. (2015). Determinants of emotion duration and underlying psychological and neural mechanisms. Emotion Review, 7(4), 330–335. https://doi.org/10.1177/1754073915590618. [GS Search].
Verduyn, P., & Lavrijsen, S. (2015). Which emotions last longest and why: The role of event importance and rumination. Motivation and Emotion, 39, 119–127. https://doi.org/10.1007/s11031-014-9445-y. [GS Search].
Verduyn, P., Van Mechelen, I., & Tuerlinckx, F. (2011). The relation between event processing and the duration of emotional experience. Emotion, 11(1), 20–28. https://doi.org/10.1037/a0021239. [GS Search].
Wang, Y., Heffernan, N. T., & Heffernan, C. (2015). Towards better affect detectors: Effect of missing skills, class features and common wrong answers. International Conference on Learning Analytics and Knowledge (LAK), 31–35. https://doi.org/10.1145/2723576.2723618. [GS Search].
Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2005). Data mining: Practical machine learning tools and techniques (Vol. 2). Elsevier. https://doi.org/10.1016/b978-0-12-374856-0.00015-8. [GS Search].
Wixon, M., & Arroyo, I. (2014). When the question is part of the answer: Examining the impact of emotion self-reports on student emotion. International Conference on User Modeling, Adaptation, and Personalization, 471–477. https://doi.org/10.1007/978-3-319-08786-3_42. [GS Search].
Yang, D., Kraut, R. E., & Rose, C. P. (2016). Exploring the effect of student confusion in massive open online courses. Journal of Educational Data Mining, 8(1), 52–83. [GS Search].
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Felipe de Morais, Diógines Goldoni, Tiago R. Kautzmann, Patricia A. Jaques
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
