Automatic Essay Evaluation in Portuguese: A Systematic Review
DOI:
https://doi.org/10.5753/rbie.2023.2869Keywords:
Essay Scoring, Content Analysis, Natural Language processingAbstract
The literature has vastly explored Automatic Essay Scoring (AES) in the last few years. The critical motivation is the possibility of reducing the human effort in scoring a large number of essays in a short period. In literature, most of the work concentrates on the English language; there is still a need for progress in Brazilian Portuguese. Thus, this works provides a Systematic Mapping Study aiming to identify Artificial Intelligence methods that support Automatic Essay Correction in Brazilian Portuguese. Furthermore, the main facts this paper brings are: (i) the methods focus on feature engineering methods instead of deep learning models; (ii) there is a prevalence of traditional metrics such as precision, coverage, and f-measure to evaluate the results; (iii) feedbacks provided by the tools have low-level of details; and (iv) there is no practical evaluation of the advancement in real-world applications.
Downloads
References
Becheikh, N., Landry, R., & Amara, N. (2006). Lessons from innovation empirical studies in the manufacturing sector: A systematic review of the literature from 1993–2003. Technovation, 26(5-6), 644-664. doi: 10.1016/j.technovation.2005.06.016 [GS Search]
Camelo, R., Justino, S., & Mello, R. (2020). Coh-Metrix PT-BR: Uma API web de análise textual para a educação. Anais dos Workshops do IX Congresso Brasileiro de Informática na Educação, 179–186. doi: 10.5753/cbie.wcbie.2020.179 [GS Search]
Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., et al. (2015). Xgboost: extreme gradient boosting. R package version 0.4-2, 1(4), 1–4. doi: 10.1145/2939672.2939785 [GS Search]
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. doi: 10.1007/BF00994018 [GS Search]
Costa, L., Oliveira, E. H. T. d., & Castro Júnior, A. (2020). Corretor Automático de Redações em Língua Portuguesa: um mapeamento sistemático de literatura. Anais do XXXI Simpósio Brasileiro de Informática na Educação (SBIE 2020), 1403–1412. doi: 10.5753/cbie.sbie.2020.1403 [GS Search]
Ferreira Mello, R., Fiorentino, G., Oliveira, H., Miranda, P., Rakovic, M., & Gasevic, D. (2022). Towards automated content analysis of rhetorical structure of written essays using sequential content-independent features in Portuguese. LAK22: 12th International Learning Analytics and Knowledge Conference, 404–414. doi: 10.1145/3506860.3506977 [GS Search]
Filho, A. H., Concatto, F., Nau, J., Prado, H. A. d., Imhof, D. O., & Ferneda, E. (2019). Imbalanced Learning Techniques for Improving the Performance of Statistical Models in Automated Essay Scoring. Procedia Computer Science, 159, 764–773. doi: 10.1016/j.procs.2019.09.235 [GS Search]
Filho, A. H., do Prado, H. A., Ferneda, E., & Nau, J. (2018). An approach to evaluate adherence to the theme and the argumentative structure of essays. Procedia Computer Science, 126, 788–797. doi: 10.1016/j.procs.2018.08.013 [GS Search]
Fonseca, E., Medeiros, I., Kamikawachi, D., & Bokan, A. (2018). Automatically Grading Brazilian Student Essays [Series Title: Lecture Notes in Computer Science]. Em A. Villavicencio, V. Moreira, A. Abad, H. Caseli, P. Gamallo, C. Ramisch, H. Gonçalo Oliveira & G. H. Paetzold (Ed.), Computational Processing of the Portuguese Language (pp. 170–179). Springer International Publishing. doi: 10.1007/978-3-319-99722-3_18 [GS Search]
Guimarães, N. S., Ferreira, A. J., Silva, R. d. C. R., de Paula, A. A., Lisboa, C. S., Magno, L., Ichiara, M. Y., & Barreto, M. L. (2022). Deduplicating records in systematic reviews: there are free, accurate automated ways to do so. Journal of Clinical Epidemiology, 152, 110–115. doi: j.jclinepi.2022.10.009 [GS Search]
Haendchen Filho, A., do Prado, H. A., Ferneda, E., & Nau, J. (2018). An approach to evaluate adherence to the theme and the argumentative structure of essays. Procedia Computer Science, 126, 788–797. doi: 10.1016/j.procs.2018.08.013 [GS Search]
Johnson, N., & Phillips, M. (2018). Rayyan for systematic reviews. Journal of Electronic Resources Librarianship, 30(1), 46–48. doi: 10.1080/1941126X.2018.1444339 [GS Search]
Kahn, J. H., Tobin, R. M., Massey, A. E., & Anderson, J. A. (2007). Measuring emotional expression with the Linguistic Inquiry and Word Count. The American journal of psychology, 120(2), 263–286. [GS Search]
Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering (rel. técn. EBSE 2007-001). Keele University e Durham University Joint Report. [GS Search]
Marinho, J., Anchiêta, R., & Moura, R. (2021). Essay-BR: a Brazilian Corpus of Essays. Anais do III Dataset Showcase Workshop, 53–64. doi: 10.5753/dsw.2021.17414 [GS Search]
McKeown, S., & Mir, Z. M. (2021). Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references. Systematic reviews, 10, 1–8. doi: 10.1186/s13643-021-01583-y [GS Search]
Mello, R. F., Fiorentino, G., Miranda, P., Oliveira, H., Rakovic, M., & Gašević, D. (2021). Towards Automatic Content Analysis of Rhetorical Structure in Brazilian College Entrance Essays [Series Title: Lecture Notes in Computer Science]. Em I. Roll, D. McNamara, S. Sosnovsky, R. Luckin & V. Dimitrova (Ed.), Artificial Intelligence in Education (pp. 162–167). Springer International Publishing. doi: 10.1007/978-3-030-78270-2_29 [GS Search]
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group*, t. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine, 151(4), 264–269. doi: 10.7326/0003-4819-151-4-200908180-00135 [GS Search]
Nau, J., Haendchen Filho, A., & Dazzi, R. L. S. (2019). Identificação e Avaliação Automática da Proposta de Intervenção em Textos Dissertativos-Argumentativos: Uma Revisão Sistemática da Literatura. Anais do Computer on the Beach, 493–501. doi: 10.4013/cld.2017.153.08 [GS Search]
Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning, 38(2), 599–620. doi: 10.1111/jcal.12635 [GS Search]
Palermo, C., & Thomson, M. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54, 255–270. doi: 10.1016/j.cedpsych.2018.07.002 [GS Search]
Papadopoulos, I., Koulouglioti, C., Lazzarino, R., & Ali, S. (2020). Enablers and barriers to the implementation of socially assistive humanoid robots in health and social care: a systematic review. BMJ open, 10(1), e033096. doi: 10.1136/bmjopen-2019-033096 [GS Search]
Safavian, S. R., & Landgrebe, D. (1991). A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics, 21(3), 660–674. doi: 10.1109/21.97458 [GS Search]
Santos, K. S., Soder, M., Marques, B. S. B., & Feltrim, V. D. (2018). Analyzing the rhetorical structure of opinion articles in the context of a Brazilian college entrance examination. Computational Processing of the Portuguese Language: 13th International Conference, PROPOR 2018, Canela, Brazil, September 24–26, 2018, Proceedings 13, 3–12. doi: 10.1007/978-3-319-99722-3_1 [GS Search]
Sousa, A., Leite, B., Rocha, G., & Lopes Cardoso, H. (2021). Cross-Lingual Annotation Projection for Argument Mining in Portuguese [Series Title: Lecture Notes in Computer Science]. Em G. Marreiros, F. S. Melo, N. Lau, H. Lopes Cardoso & L. P. Reis (Ed.), Progress in Artificial Intelligence (pp. 752–765). Springer International Publishing. doi: 10.1007/978-3-030-86230-5_59 [GS Search]
Tang, J., & Rich, C. S. (2017). Automated writing evaluation in an EFL setting Lessons from China. JALT CALL Journal, 13(2), 117--146. doi: 10.29140/jaltcall.v13n2.215 [GS Search]
Ware, P. (2014). Feedback for Adolescent Writers in the English Classroom. Writing & Pedagogy, 6(2). doi: 10.1558/wap.v6i2.223 [GS Search]
Tang, J., & Rich, C. S. (2017). Automated writing evaluation in an EFL setting Lessons from China. JALT CALL Journal, 13(2), 117--146. doi: 10.29140/jaltcall.v13n2.215 [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Tiago Barbosa de Lima, Ingrid Luana Almeida da Silva, Elyda Laisa Soares Xavier Freitas, Rafael Ferreira Mello
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
