Comparative Study between Models that Estimate Student’s Skill in Virtual Programming Environment
DOI:
https://doi.org/10.5753/rbie.2020.28.0.776Keywords:
Skil, IRT, Elo, M-ERSAbstract
The number of online platforms offering programming exercises is increasing, where students submit exercise resolutions and receive automatic feedback from the system, without human intervention. These environments allow the recording of many aspects of the submissions and, thus, the educational evaluation models can be used to infer the skills worked in each solution. In this paper we present a comparative analysis of three models that estimate student’s skill: Elo, Item Response Theory (IRT) and M-ERS (Multidimensional Extension of the ERS). Elo was developed to classify chess players, through their game history, but it was adapted to estimate the student’s skill through the history of problem submissions. The IRT estimates the skill through a set of answers given to a set of items, there are some IRT models that vary according to the type of response. M-ERS is an adaptation of Elo and IRT that combines the two models and tracks the multiple skills of students. The Elo models, 2-parameter IRT, gradual response IRT and M-ERS were applied to a database provided by an Online Judge platform. The results obtained point out differences between the models regarding the estimated skills, differences that are believed to be related to the way in which each model estimates the parameters.
Downloads
References
Andrade, D., & Justino, G. (2007). Software para avaliação de aprendizagem utilizando a teoria da resposta ao item. In XIII Workshop sobre Informática na Escola. Rio de Janeiro–RJ. [DOI:10.5753/cbie.wie.2007] [GS Search]
Andrade, D., Tavares, H. R., & Cunha, V. R. (2000). Teoria da resposta ao item: conceitos e aplicações. São Paulo: ABE - Associação Brasileira de Estatística. [GS Search]
Araujo, E., Andrade, D., & Bortolotti, S. (2009). Teoria da resposta ao item. Revista da Escola de Enfermagem da USP, 43, 1000-1008. [DOI: 10.1590/S0080-62342009000500003] [GS Search]
Baker, F. (2001). The basics of item response theory (2nd ed.). Washington: ERIC. [GS Search]
Bez, J. L., Tonin, N. A., & Rodegher, P. R. (2011). URI Online Judge. Retrieved from https://www.urionlinejudge.com.br/ [GS Search]
Bez, J. L., Tonin, N. A., & Rodegheri, P. R. (2014). URI Online Judge Academic: A tool for algorithms and programming classes. In 9th international conference on computer science education (p. 149-152). [DOI: 10.5753/wei.2015.10235] [GS Search]
Braga, B. (2015). Teoria da resposta ao item: o uso do modelo de Samejima como proposta de correção para itens discursivos. UnB, Brasília. [GS Search]
Chalmers, R. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1-29. [DOI: 10.18637/jss.v048.i06] [GS Search]
Elo, A. (1978). The rating of chessplayers, past and present. London: Batsford. [GS Search]
França, L. (2020). Competências e habilidades no ensino: o que são e como aplicá-las? Retrieved from https://www.somospar.com.br/competencias-e-habilidades/ [GS Search]
Moreira, G. L., Holanda, W., Coutinho, J. C., & Chagas, F. (2018). Desafios na aprendizagem de programação introdutória em cursos de TI da UFERSA, campus Pau dos Ferros: um estudo exploratório. Anais do Encontro de Computação do Oeste Potiguar ECOP/UFERSA(2). [GS Search]
Moreira Junior, F. (2010). Aplicações da teoria da resposta ao item (TRI) no Brasil. Revista Brasileira de Biometria, 28(4), 137–70. [GS Search]
Nojosa, R. T. (2002). Teoria da Resposta ao Item (TRI): modelos multidimensionais. Estudos em Avaliação Educacional(25), 123–166. [DOI: 10.18222/eae02520022193] [GS Search]
Oliveira, L. (2017). O desempenho em matemática do enem de 2012 em Luis Eduardo Magalhães (BA), na teoria de resposta ao item. UFT, Arraias. [GS Search]
Park, J., Cornillie, F., Van der Maas, H., & Van Den, N. (2019). A multidimensional IRT approach for dynamically monitoring ability growth in computerized practice environments. Frontiers in psychology, 10. [DOI: 10.3389/fpsyg.2019.00620] [GS Search]
Pasquali, L. (2018). Tri–teoria de resposta ao item: Teoria, procedimentos e aplicações. Curitiba: Appris. [GS Search]
Pelánek, R. (2016). Applications of the elo rating system in adaptive educational systems. Computers Education, 98, 169-179. [DOI: 10.1016/j.compedu.2016.03.017] [GS Search]
Perrenoud, P., & Magne, B. (1999). Construir as competências desde a escola. Porto Alegre:Artmed.[GS Search]
Prisco, A., Santos, R., Botelho, S., Tonin, N., & Bez, J. (2018). A multidimensional Elo model for matching learning objects. In 2018 IEEE Frontiers in Education Conference (FIE). [GS Search]
Reckase, M. D. (2006). Multidimensional item response theory. Handbook of statistics, 26, 607–642. [DOI: 10.1016/S0169-7161(06)26018-8] [GS Search]
Robins, A. (2010). Learning edge momentum: A new account of outcomes in CS1. Computer Science Education, 20(1), 37–71. [DOI: 10.1080/08993401003612167] [GS Search]
Soares, T. M., Souza, R. C., & Pereira, V. R. (2004). Métodos alternativos no critério Brasil para construção de indicadores sócio-econômicos: Teoria da resposta ao item. XXXVI Simpósio Brasileiro de Pesquisa Operacional, 35. [GS Search]
Tavares, C. (2014). A teoria de resposta ao item na avaliação em larga escala: Um estudo sobre o exame nacional de acesso ao mestrado profissional em matemática em rede nacional-profmat. IMPA, Rio de Janeiro. [GS Search]
Wasik, S., Antczak, M., Badura, J., Laskowski, A., & Sternal, T. (2018). A survey on online judge systems and their applications. ACM Computing Surveys (CSUR), 51(1), 1–34. [DOI: 10.1145/3143560] [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Fabiana Zaffalon, André Prisco, Ricardo Souza, Jean Luca Bez, Neilor Tonin, Rafael Penna, Silvia Botelho
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
