Essay-BR: a Brazilian Corpus to Automatic Essay Scoring Task
DOI:
https://doi.org/10.5753/jidm.2022.2340Keywords:
Automated Essay Evaluation, Textual Databases, Natural Language ProcessingAbstract
Automatic Essay Scoring (AES) is the computer technology that evaluates and scores the written essays, aiming to provide computational models to grade essays automatically or with minimal human involvement. While there are several AES studies in a variety of languages, few of them are focused on the Portuguese language. The main reason is the lack of a corpus with manually graded essays. In order to bridge this gap, in this paper we extended a corpus of essays written by Brazilian high school students in an online platform. All of the essays are argumentative and were scored across five competences by experts. Moreover, we conducted an experiment with the extended corpus to show some challenges posed by the Portuguese language. The corpus are publicly available at https://github.com/lplnufpi/essay-br.
Downloads
References
Amorim, E., Cançado, M., and Veloso, A. Automated essay scoring in the presence of biased ratings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, pp. 229–237, 2018.
Amorim, E. and Veloso, A. A multi-aspect analysis of automatic essay scoring for Brazilian Portuguese. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Valencia, Spain, pp. 94–102, 2017.
Bazelato, B. and Amorim, E. A bayesian classifier to automatic correction of portuguese essays. In XVIII Conferência Internacional sobre Informática na Educação. Nuevas Ideas en Informática Educativa, Porto Alegre, Brazil, pp. 779–782, 2013.
Beigman Klebanov, B., Flor, M., and Gyawali, B. Topicality-based indices for essay scoring. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, San Diego, CA, pp. 63–72, 2016.
Beigman Klebanov, B. and Madnani, N. Automated evaluation of writing – 50 years and counting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, pp. 7796–7810, 2020.
Bird, S. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions. Association for Computational Linguistics, Sydney, Australia, pp. 69–72, 2006.
Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., and Chodorow, M. Toefl11: A corpus of non-native english. ETS Research Report Series 2013 (2): i–15, 2013.
Cohen, J. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70 (4): 213–220, 1968.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp. 4171–4186, 2019.
Dikli, S. An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment 5 (1): 1–36, 2006.
Dong, F., Zhang, Y., and Yang, J. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Association for Computational Linguistics, Vancouver, Canada, pp. 153–162, 2017.
Farra, N., Somasundaran, S., and Burstein, J. Scoring persuasive essays using opinions and their targets. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Denver, Colorado, pp. 64–74, 2015.
Fonseca, E., Medeiros, I., Kamikawachi, D., and Bokan, A. Automatically grading brazilian student essays. In Proceedings of the 13th International Conference on Computational Processing of the Portuguese Language. Springer International Publishing, Canela, Brazil, pp. 170–179, 2018.
Ke, Z. and Ng, V. Automated essay scoring: a survey of the state of the art. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, Macao, China, pp. 6300–6308, 2019.
Marinho, J. C., Anchiêta, R. T., and Moura, R. S. Essay-br: a brazilian corpus of essays. In XXXIV Simpósio Brasileiro de Banco de Dados: Dataset Showcase Workshop, SBBD 2021 Companion. SBC, Online, pp. 53–64, 2021.
Mayfield, E. and Black, A. W. Should you fine-tune BERT for automated essay scoring? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Online, pp. 151–162, 2020.
Nguyen, H. V. and Litman, D. J. Argument mining for improving the automated scoring of persuasive essays. In Thirty-Second AAAI Conference on Artificial Intelligence. AAAI Press, Louisiana, USA, pp. 5892–5899, 2018.
Page, E. B. The imminence of... grading essays by computer. The Phi Delta Kappan 47 (5): 238–243, 1966.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research vol. 12, pp. 2825–2830, 2011.
Scarton, C., Gasperin, C., and Aluisio, S. Revisiting the readability assessment of texts in portuguese. In Proceedings of the 12th Ibero-American Conference on Artificial Intelligence. Springer, Bahía Blanca, Argentina, pp. 306–315, 2010.
Shermis, M. D. and Barrera, F. D. Exit assessments: Evaluating writing ability through automated essay scoring, 2002.
Shermis, M. D. and Burstein, J. Handbook of automated essay evaluation: Current applications and new directions. Routledge, 2013.
Stab, C. and Gurevych, I. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics, Dublin, Ireland, pp. 1501–1510, 2014.
Sylviane Granger, Estelle Dagneaux, F. M. and Paquot, M. International Corpus of Learner English (Version 2). UCL Presses de Louvain, 2009.
Taghipour, K. and Ng, H. T. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pp. 1882–1891, 2016.
Vajjala, S. Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education 28 (1): 79–105, 2018.
Williamson, D. M. A framework for Implementing Automated Scoring. In Annual Meeting of the American Educational Research Association and the National Council on Measurement in Education. San Diego, CA, pp. 39, 2009.
Yannakoudakis, H., Briscoe, T., and Medlock, B. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, pp. 180–189, 2011.
Yannakoudakis, H. and Cummins, R. Evaluating the performance of automated text scoring systems. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Denver, Colorado, pp. 213–223, 2015.
