Emotion-Aware Framework for Equitable Facial Expression Synthesis in Interactive Educational Systems

Diego Addan Gonçalves; Eduardo Todt

doi:10.5753/jis.2026.6580

Authors

Diego Addan Gonçalves Federal University of Paraná https://orcid.org/0000-0003-3355-3966
Eduardo Todt Federal University of Paraná https://orcid.org/0000-0001-6045-1274

DOI:

https://doi.org/10.5753/jis.2026.6580

Keywords:

Emotions in Interactive Systems, Facial Expression Synthesis, Affective Computing, User Experience (UX), Cultural Bias Mitigation, Educational Technology, Human-Computer Interaction (HCI)

Abstract

Emotions are central to interactive systems, shaping user experience, decision-making, and engagement. In educational contexts, emotionally expressive avatars can foster empathy, motivation, and social belonging, but existing solutions face critical limitations. This article proposes an emotion-aware multidimensional framework that integrates technical, pedagogical, sociocultural, and operational perspectives to guide the equitable design of interactive educational systems. Grounded in a systematic review of 127 studies (2014–2024), our results show that current approaches face three key challenges: (1) realism-accessibility trade-offs (e.g., diffusion models’ F1=0.91 vs. GANs’ 34ms latency), (2) cultural bias in emotion recognition (89% Western dominance reduced to 20% using our adaptation protocols), and (3) limited pedagogical integration of affective features (with gains of 23% in retention when aligned with Bloom’s Taxonomy). The review highlights fragmentation across technical, pedagogical, and cultural domains, underscoring the need for integration. To address this, we propose an Emotion-Aware Multidimensional Framework that unifies these perspectives into actionable design and evaluation protocols. By situating emotions as a core dimension of system design, the study contributes not only to educational applications but also to the broader field of interactive systems where affective engagement is critical.

Downloads

Download data is not yet available.

References

Baylor, A. L. (2019). The impact of pedagogical agent gesturing in multimedia learning environments: A meta-analysis. Educational Research Review, 28. DOI: https://doi.org/10.1016/j.edurev.2019.05.002.

Baylor, A. L., Kim, Y., and Lee, S. (2023). Pedagogical agents in special education: A systematic review of affective computing applications. Journal of Educational Computing Research, 61(4):783–810. DOI: https://doi.org/10.1177/07356331231154420.

Braun, V. and Clarke, V. (2022). Thematic analysis: A practical guide. Sage.

Buolamwini, J. and Gebru, T. (2023). Intersectional ai: Understanding and addressing the social impacts of artificial intelligence. AI & Society, 38(2):1–12. DOI: https://doi.org/10.1007/s00146-022-01496-3.

Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., and Tong, X. (2023). Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):346–360. DOI: https://doi.org/10.1109/TPAMI.2021.3130384.

Gusenbauer, M. and Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? evaluating retrieval qualities of google scholar, pubmed, and 26 other resources. Research Synthesis Methods, 11(2):181–217. DOI: https://doi.org/10.1002/jrsm.1378.

Johnson, W. L. and Lester, J. C. (2021). Digital human pedagogical agents to support learning. International Journal of Artificial Intelligence in Education, 31(4):723–755. DOI: https://doi.org/10.1007/s40593-021-00256-7.

Johnson, W. L., Lester, J. C., and Lee, J. (2023). Affective pedagogical agents in language learning: A meta-analysis of emotional expression effects. Computer Assisted Language Learning, 36(5):912–940. DOI: https://doi.org/10.1080/09588221.2023.2181382.

Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34:852–863. StyleGAN3 paper.

Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report. EBSE Technical Report EBSE-2007-01.

Kolestra, J., van der Meij, H., and de Jong, T. (2016). Personalized affective feedback for adaptive learning systems. Computers & Education, 101:85–98. DOI: https://doi.org/10.1016/j.compedu.2016.06.004.

Kärkkäinen, K. and Joo, J. (2021). Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint. arXiv:1908.04913. DOI: https://doi.org/10.48550/arXiv.1908.04913.

Mollahosseini, A., Hasani, B., and Mahoor, M. H. (2019). Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18–31. DOI: https://doi.org/10.1109/TAFFC.2017.2740923.

Ortega-Ochoa, E., Arguedas, M., and Daradoumis, T. (2024). Empathic pedagogical conversational agents: A systematic literature review. British Journal of Educational Technology, 55:886–909. DOI: https://doi.org/10.1111/bjet.13413.

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., and Moher, D. (2021). The prisma 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1):1–11. DOI: https://doi.org/10.1186/s13643-021-01626-4.

Pereira, R., Darin, T., and Silveira, M. S. (2024). Grandihc-br: Grand research challenges in human-computer interaction in brazil for 2025-2035. Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems (IHC ’24), pages 1–24. DOI: https://doi.org/10.1145/3702038.3702061.

Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint. arXiv:2307.01952. DOI: https://doi.org/10.48550/arXiv.2307.01952.

Ruiz, N., Cheng, Y., and Liu, Y. (2024). Real-time affective adaptation in intelligent tutoring systems: A design-based research approach. IEEE Transactions on Learning Technologies, 17(2):456–473. DOI: https://doi.org/10.1109/TLT.2023.3298741.

Ruiz, N., Liu, Y., Jourabloo, A., and Cheng, Y. (2023). Learning bias-invariant facial representations for domain generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3723–3736. DOI: https://doi.org/10.1109/TPAMI.2022.3194378.

Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. (2023). Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):1485–1497. DOI: https://doi.org/10.1109/TPAMI.2022.3204461.

Septiana, I., Mutijarsa, K., Putro, B. L., and Rosmansyah, Y. (2024). Emotion-related pedagogical agent: A systematic literature review. IEEE Access, 12. DOI: https://doi.org/10.1109/ACCESS.2024.3374376.

Valstar, M. F., Pantic, M., and Cohn, J. F. (2017). Facs-based facial expression recognition: A comprehensive review. IEEE Transactions on Affective Computing, 8(2):156–176. DOI: https://doi.org/10.1109/TAFFC.2017.2705084.

Vieira, L. N., O’Hagan, M., and O’Sullivan, C. (2021). Understanding the societal impacts of machine translation: A critical review of the literature. Translation Spaces, 10(2):171–200. DOI: https://doi.org/10.1075/ts.21002.vie.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in software engineering. Springer.

Zhang, Y., Zhang, H., Cun, X., Shen, X., Guo, Y., Shan, Y., and Wang, F. (2023). A survey of facial animation generation and editing with deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5591–5610. DOI: https://doi.org/10.1109/TPAMI.2022.3204461.