Which Feedback Is More Effective? A Comparative Study on Teachers' Evaluation of Feedback Models in Essay Writing
DOI:
https://doi.org/10.5753/rbie.2026.7395Keywords:
Educational Feedback, Large Language Models (LLMs), ENEM Essay, Feedback LiteracyAbstract
Feedback is a key factor in improving the writing skills of students, but providing it at a large scale remains a significant challenge. Large Language Models (LLMs) emerge as a promising solution; however, the pedagogical quality of the automatically generated feedback requires further investigation. This study examined how effective teachers perceived feedback texts generated by an LLM when the prompts were designed to reflect different aspects of feedback theory. To this end, 450 feedback texts were generated for 30 essays and evaluated by five teachers with varying feedback literacy profiles. The results showed that teachers' preferences for feedback models were strongly shaped by their profiles, indicating that there was no single universally "best" feedback model. From a learning analytics perspective, these findings demonstrate how combining theory-informed prompt design with evaluations from diverse teacher profiles can generate actionable insights for the development of scalable, AI-powered feedback systems. The study contributes to learning by showing that the effectiveness of LLM-based feedback depends on personalization not only for students but also for teachers, highlighting new directions for designing adaptive, equitable, and context-sensitive feedback analytics.
Downloads
Referências
Alcock, S., Rienties, B., Aristeidou, M., & Kouadri Mostéfaoui, S. (2024). How do visualizations and automated personalized feedback engage professional learners in a learning analytics dashboard? Proceedings of the 14th Learning Analytics and Knowledge Conference, 316–325. https://doi.org/10.1145/3636555.3636886 [GS Search].
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354 [GS Search].
Carless, D., & Winstone, N. (2023). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in higher education, 28(1), 150–163. https://doi.org/10.1080/13562517.2020.1782372 [GS Search].
Carvalho, R., Lins, L. F., Rodrigues, L., Miranda, P., Oliveira, H., Cordeiro, T., Bittencourt, I. I., Isotani, S., & Mello, R. F. (2024). Exploring NLP and embedding for automatic essay scoring in the Portuguese. International Conference on Artificial Intelligence in Education, 228–233. https://doi.org/10.1007/978-3-031-64315-6_18 [GS Search].
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gaševic, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2, 100027. https://doi.org/10.1016/j.caeai.2021.100027 [GS Search].
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., & Gaševic, D. (2020). How good is my feedback? a content analysis of written feedback. Proceedings of the 10th International Conference on Learning Analytics and Knowledge. https://doi.org/10.1145/3375462.3375477 [GS Search].
Cavalcanti, A. P., Mello, R. F. L., Rolim, V., André, M., Freitas, F., & Gaševic, D. (2019). An analysis of the use of good feedback practices in online learning courses. 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT), 2161, 153–157. https://doi.org/10.1109/ICALT.2019.00061 [GS Search].
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104 [GS Search].
Cole, R. (2024). Inter-rater reliability methods in qualitative case study research. Sociological Methods & Research, 53(4), 1944–1975. https://doi.org/10.1177/00491241231156971 [GS Search].
Dawson, S., Pardo, A., Salehian Kia, F., & Panadero, E. (2023). An integrated model of feedback and assessment: From fine grained to holistic programmatic review. LAK23: 13th International Learning Analytics and Knowledge Conference, 579–584. https://doi.org/10.1145/3576050.3576074 [GS Search].
Evans, C. (2013). Making sense of assessment feedback in higher education. Review of educational research, 83(1), 70–120. https://doi.org/10.3102/0034654312474350 [GS Search].
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5), 378. https://doi.org/10.1037/h0031619 [GS Search].
Galhardi, L., Herculano, M. F., Rodrigues, L., Miranda, P., Oliveira, H., Cordeiro, T., Bittencourt, I. I., Isotani, S., & Mello, R. F. (2024). Contextual features for automatic essay scoring in Portuguese. International Conference on Artificial Intelligence in Education, 270–282. https://doi.org/10.1007/978-3-031-64315-6_23 [GS Search].
Gibbs, G., & Simpson, C. (2005). Conditions under which assessment supports students' learning [Link]. Learning and teaching in higher education, (1), 3–31. [GS Search].
Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. [GS Search].
Hattie, J., & Gan, M. (2011). Instruction based on feedback. In Handbook of research on learning and instruction (pp. 263–285). Routledge. https://doi.org/10.4324/9780203839089-22 [GS Search].
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81–112. https://doi.org/10.3102/003465430298487 [GS Search].
Hutt, S., DePiro, A., Wang, J., Rhodes, S., Baker, R. S., Hieb, G., Sethuraman, S., Ocumpaugh, J., & Mills, C. (2024). Feedback on feedback: Comparing classic natural language processing and generative AI to evaluate peer feedback. Proceedings of the 14th Learning Analytics and Knowledge Conference, 55–65. https://doi.org/10.1145/3636555.3636850 [GS Search].
Jansen, T., Horbach, A., & Meyer, J. (2025). Feedback from generative AI: Correlates of student engagement in text revision from 655 classes from primary and secondary school. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 831–836. https://doi.org/10.1145/3706468.3706494 [GS Search].
Kucia, F. J., Chakraborty, A., & Wróblewska, A. (2026). LLM essay scoring under holistic and analytic rubrics: Prompt effects and bias. arXiv preprint arXiv:2604.00259. https://doi.org/10.48550/arXiv.2604.00259 [GS Search].
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310 [GS Search].
Lee, I., Karaca, M., & Inan, S. (2023). The development and validation of a scale on l2 writing teacher feedback literacy. Assessing Writing, 57, 100743. https://doi.org/10.1016/j.asw.2023.100743 [GS Search].
Lin, J., Dai, W., Lim, L.-A., Tsai, Y.-S., Mello, R. F., Khosravi, H., Gasevic, D., & Chen, G. (2023). Learner-centred analytics of feedback content in higher education. LAK23: 13th international learning analytics and knowledge conference, 100–110. https://doi.org/10.1145/3576050.3576064 [GS Search].
Marcoci, A., Webb, M. E., Rowe, L., Barnett, A., Primoratz, T., Kruger, A., Karvetski, C. W., Stone, B., Diamond, M. L., Saletta, M., van Gelder, T., Tetlock, P. E., & Dennis, S. (2024). Validating a forced-choice method for eliciting quality-of-reasoning judgments. Behavior Research Methods, 56(5), 4958–4973. https://doi.org/10.3758/s13428-023-02234-x [GS Search].
Marinho, J. C., Anchiêta, R. T., & Moura, R. S. (2021). Essay-BR: a Brazilian Corpus of Essays. arXiv preprint arXiv:2105.09081. https://doi.org/10.5753/dsw.2021.17414 [GS Search].
Mello, R. F., Anthony, L., Lobo, J., Ribeiro, F. G. C., Xavier, C., Costa, N. T., Gasevic, D., & Rodrigues, L. (2025). Empowering equitable learning with LLMs: Enhancing writing skills in low-resource contexts. European Conference on Technology Enhanced Learning, 183–197. https://doi.org/10.1007/978-3-032-03870-8_13 [GS Search].
Mello, R. F., Freitas, E., Cabral, L., Pereira, F. D., Rodrigues, L., Rakovic, M., Raniel, J., & Gaševic, D. (2024). Words of wisdom: A journey through the realm of natural language processing for learning analytics–a systematic literature review. Journal of Learning Analytics, 11(3), 82–105. https://doi.org/10.18608/jla.2024.8403 [GS Search].
Monteiro, E., & Barreto, R. (2022). Um método baseado na teoria da resposta ao item para avaliação e feedback automático no contexto do ENEM. Simpósio Brasileiro de Informática na Educação (SBIE), 255–266. https://doi.org/10.5753/rbie.2021.29.0.746 [GS Search].
Nguyen, H., & Park, S. (2025). Providing automated feedback on formative science assessments: Uses of multimodal large language models. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 803–809. https://doi.org/10.1145/3706468.3706480 [GS Search].
Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090 [GS Search].
Oliveira, H., Mello, R. F., Miranda, P., Alexandre, B., Cordeiro, T., Bittencourt, I. I., & Isotani, S. (2023). Classificação ou regressão? Avaliando coesão textual em redações no contexto do ENEM. Simpósio Brasileiro de Informática na Educação (SBIE), 1226–1237. https://doi.org/10.5753/sbie.2023.234516 [GS Search].
Oliveira, H., Mello, R. F., Miranda, P., Batista, H., Silva Filho, M. W., Cordeiro, T., Bittencourt, I. I., & Isotani, S. (2025). A benchmark dataset of narrative student essays with multicompetency grades for automatic essay scoring in Brazilian Portuguese. Data in Brief, 60, 111526. https://doi.org/10.1016/j.dib.2025.111526 [GS Search].
Pardo, A., Jovanovic, J., Dawson, S., Gaševic, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/bjet.12592 [GS Search].
Pimentel, E. P., Silva, L. D. L., Takeuti, R. H., & Braga, J. C. (2025). Abordagem com LLM para geração automatizada de feedback no ensino de programação de computadores. Simpósio Brasileiro de Informática na Educação (SBIE), 590–602. https://doi.org/10.5753/sbie.2025.12543 [GS Search].
Robertson, J., & Kaptein, M. (2016). An introduction to modern statistical methods in hci. In Modern statistical methods for hci (pp. 1–14). Springer. https://doi.org/10.1007/978-3-319-26633-6_1 [GS Search].
Rüdian, S., Podelo, J., Kužílek, J., & Pinkwart, N. (2025). Feedback on feedback: Student's perceptions for feedback from teachers and few-shot LLMs. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 82–92. https://doi.org/10.1145/3706468.3706479 [GS Search].
Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2), 119–144. https://doi.org/10.1007/BF00117714 [GS Search].
Shermis, M. D., & Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge. [GS Search].
Shibani, A., Knight, S., & Buckingham Shum, S. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Proceedings of the 9th international conference on learning analytics & knowledge, 210–219. https://doi.org/10.1145/3303772.3303785 [GS Search].
Shin, I., Hwang, S. B., Yoo, Y. J., Bae, S., & Kim, R. Y. (2025). Comparing student preferences for AI-generated and peer-generated feedback in AI-driven formative peer assessment. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 159–169. https://doi.org/10.1145/3706468.3706488 [GS Search].
Singh, A., Brooks, C., Wang, X., Li, W., Kim, J., & Wilson, D. (2024). Bridging learnersourcing and AI: Exploring the dynamics of student-AI collaborative feedback generation. Proceedings of the 14th Learning Analytics and Knowledge Conference, 742–748. https://doi.org/10.1145/3636555.3636853 [GS Search].
Van Campenhout, R., Kimball, M., Clark, M., Dittel, J. S., Jerome, B., & Johnson, B. G. (2024). An investigation of automatically generated feedback on student behavior and learning. Proceedings of the 14th Learning Analytics and Knowledge Conference, 850–856. https://doi.org/10.1145/3636555.3636901 [GS Search].
Watson, P., & Petrie, A. (2010). Method agreement analysis: A review of correct methodology. Theriogenology, 73(9), 1167–1179. https://doi.org/10.1016/j.theriogenology.2010.01.003 [GS Search].
Watts, F. M., Dood, A. J., & Shultz, G. V. (2023). Automated, content-focused feedback for a writing-to-learn assignment in an undergraduate organic chemistry course. LAK23: 13th international learning analytics and knowledge conference, 531–537. https://doi.org/10.1145/3576050.3576053 [GS Search].
Xavier, C., Costa, N. T., Valdo, A. K., Alves, G., Rodrigues, L., Rodrigues, L. F., Silva, M., Neto, R., Falcão, T. P., Gasevic, D., & Mello, R. F. (2025). Human teacher vs. LLM-generated feedback in secondary education: A comparative study on student perceptions. European Conference on Technology Enhanced Learning, 534–548. https://doi.org/10.1007/978-3-032-03870-8_36 [GS Search].
Xiao, C., Ma, W., Song, Q., Xu, S. X., Zhang, K., Wang, Y., & Fu, Q. (2025). Human-AI collaborative essay scoring: A dual-process framework with LLMs. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 293–305. https://doi.org/10.1145/3706468.3706507 [GS Search].
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., & Gaševic, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1), 90–112. https://doi.org/10.1111/bjet.13370 [GS Search].
Yun, J., Nie, A., Brunskill, E., & Demszky, D. (2025). Exploring the benefit of customizing feedback interventions for educators and students with offline contextual multi-armed bandits. Proceedings of the 15th International Learning Analytics and Knowledge Conference, 944–949. https://doi.org/10.1145/3706468.3706551 [GS Search].
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2026 Anderson Pinheiro Cavalcanti, Luiz Rodrigues, Cleon Xavier, Newarney Torrezão da Costa, Fabiola Ribeiro, Luiz Felipe Bagnhuk Silva, Weslei Chaleghi de Melo, Letícia Santana Stacciarini, Guilherme Weber, Evelyn Vieira, Rafael Ferreira Leite de Mello, Dragan Gasevic
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
