Using code atributes to classify the difficulty of programming tasks in online judges
DOI:
https://doi.org/10.5753/rbie.2021.29.0.1137Keywords:
Difficulty, Classification, Programming tasks, Code atributes, Introductory programmingAbstract
In introductory programming classes, online judges are used as a tool for preparing assessments through a random draw of programming questions. For the random draw of questions to be balanced, questions should be classified according to their difficulty or ease. Thus, this work presents two approaches to classify programming questions by using attributes automatically extracted from solution codes for the questions. In total, 404 questions with implementation in Python were classified, which were used in evaluations of classes of introduction to programming taught between 2017 and 2019. Both approaches presented good results for dichotomous classification of programming questions.
Downloads
References
Araujo, A., Filho, D., Oliveira, E., Carvalho, L., Pereira, F., & Oliveira, D. (2021). Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do simpósio brasileiro de educação em computação (pp. 123–131). Porto Alegre, RS, Brasil: SBC. doi: 10.5753/educomp.2021.14478 [GS Search]
Bez, J. L., Ferreira, C. E., & Tonin, N. (2013). Uri online judge academic: A tool for professors. In Proceedings of the 2013 international conference on advanced ict and education (pp.744–747). Hainan, China: Atlantis Press. doi: 10.5753/educomp.2021.14478 [GS Search]
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). Smote: Synthetic minority over-sampling technique. J. Artif. Int. Res., 16(1), 321—357. doi: 10.1613/jair.953 [GS Search]
Chen, T., & Guestrin, C. (2016, Aug). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and DataMining. doi: 10.1145/2939672.2939785 [GS Search]
da Rocha Braz, A. C., de Carvalho, L. S. G., de Oliveira, E. H. T., de Oliveira, D. B. F., Pereira,F. D., Bittencourt, R. A., & Santana, B. L. (2021). Validação e análise de um inventário de conceitos sobre programação introdutória. In Anais estendidos do simpósio brasileiro deeducação em computação (pp. 27–28). [GS Search]
de Freitas Júnior, H. B., Pereira, F. D., de Oliveira, E. H. T., de Oliveira, D. B. F., & de Carvalho,L. S. G. (2020). Recomendação automática de problemas em juízes online usando pro-cessamento de linguagem natural e análise dirigida aos dados. In Anais do XXXI simpósio brasileiro de informática na educação (pp. 1152–1161). doi: 10.5753/cbie.sbie.2020.1152 [GS Search]
Denny, P., Cukierman, D., & Bhaskar, J. (2015). Measuring the effect of inventing practice exercises on learning in an introductory programming course. In Proceedings of the 15th koli calling conference on computing education research (p. 13–22). New York, NY, USA:Association for Computing Machinery. doi: 10.1145/2828959.2828967 [GS Search]
de Oliveira, J., Salem, F., de Oliveira, E. H. T., Oliveira, D. B. F., de Carvalho, L. S. G., & Pereira,F. D. (2020). Os estudantes leem as mensagens de feedback estendido exibidas em juízes online? In Anais do XXXI Simpósio Brasileiro de Informática na Educação (pp. 1723–1732). doi: 10.5753/cbie.sbie.2020.1723 [GS Search]
Effenberger, T.,ˇCechák, J., & Pelánek, R. (2019). Measuring difficulty of introductory program-ming tasks. In Proceedings of the sixth (2019) acm conference on learning @ scale.NewYork, NY, USA: Association for Computing Machinery. doi: 10.1145/3330430.3333641 [GS Search]
Elnaffar, S. (2016). Using software metrics to predict the difficulty of code writing questions. In 2016 ieee global engineering education conference (educon)(pp. 513–518). doi: 10.1109/EDUCON.2016.7474601 [GS Search]
Elo, A. E. (1978). The rating of chessplayers, past and present. Arco Pub. [GS Search]
Fix, E., & Hodges Jr, J. L. (1952). Discriminatory analysis-nonparametric discrimination: Small sample performance (Technical Report Project 21-49-004 No. 11). Randolf Field, Texas: USAF School of Aviation Medicine. [GS Search]
Fonseca, S. C., Pereira, F. D., Oliveira, E. H., Oliveira, D. B., Carvalho, L. S., & Cristea, A. I.(2020). Automatic subject-based contextualisation of programming assignment lists. EDM. [GS Search]
Francisco, R., Júnior, C., & Ambrósio, A. (2016). Juiz online no ensino de programação introdutória - uma revisão sistemática da literatura. Brazilian Symposium on Computersin Education (Simpósio Brasileiro de Informática na Educação - SBIE), 27(1), 11. doi: 10.5753/cbie.sbie.2016.11 [GS Search]
Francisco, R. E., & Ambrosio, A. P. (2015, jun). Mining an online judge system to support introductory computer programming teaching. In SMLIR: Workshop on tools and technologies in statistics, machine learning and information retrieval for educational data mining (pp.93–98). [GS Search]
Francisco, R. E., Ambrósio, A. P. L., Junior, C. X. P., & Fernandes, M. A. (2018). Juiz online no ensino de CS1 – lições aprendidas e proposta de uma ferramenta. Revista Brasileira deInformática na Educação, 26(03), 163. doi: 10.5753/rbie.2018.26.03.163 [GS Search]
Galvão, L., Fernandes, D., & Gadelha, B. (2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. Brazilian Symposium on Computersin Education (Simpósio Brasileiro de Informática na Educação - SBIE), 27(1), 140. doi: 10.5753/cbie.sbie.2016.140 [GS Search]
Halstead, M. H. (1977). Elements of software science (operating and programming systems series).
INEP (2017). Relatório síntese de área – Ciência da Computação. Instituto Nacional de Estudose Pesquisas Educacionais Anísio Teixeira.
Joy, M., & Luck, M. (1999). Plagiarism in programming assignments. IEEE Transactions oneducation, 42(2), 129–133. doi: 10.1109/13.762946 [GS Search]
Llana, L., Martin-Martin, E., & Pareja-Flores, C. (2012). Flop, a free laboratory of programming. In Proceedings of the 12th koli calling international conference on computing educationresearch (p. 93–99). New York, NY, USA: Association for Computing Machinery. doi: 10.1145/2401796.2401807 [GS Search]
McCabe, T. J. (1976). A complexity measure. IEEE Transactions on Software Engineering, SE–2(4), 308–320. doi: 10.1109/TSE.1976.233837 [GS Search]
Meisalo, V., Sutinen, E., & Torvinen, S. (2004). Classification of exercises in a virtual programming course. In 34th annual frontiers in education, 2004. fie 2004. (pp. S3D–1). doi: 10.1109/FIE.2004.1408764 [GS Search]
Neves, A., Oliveira, M., França, H., Lopes, M., Reblin, L., & Oliveira, E. (2017). Pcodigo ii: O sistema de diagnóstico da aprendizagem de programação por métricas de software. In Anais dos Workshops do Congresso Brasileiro de Informática na Educação, 6(1), 339. doi: 10.5753/cbie.wcbie.2017.339 [GS Search]
Pelánek, R. (2016). Applications of the elo rating system in adaptive educational systems. Computers & Education, 98, 169–179. doi: 10.1016/j.compedu.2016.03.017 [GS Search]
Pereira, F., Junior, H., Rodriquez, L., Toda, A., Oliveira, E., Cristea, A., ... Isotani, S. (2021). A recommender system based on effort: Towards minimising negative affects and maximising achievement in cs1 learning. In Intelligent tutoring systems: 17th international conference, its 2021, virtual event, june 7–11, 2021, proceedings (p. 466). doi: 10.1007/978-3-030-80421-351 [GS Search]
Pereira, F. D., Oliveira, E., Cristea, A., Fernandes, D., Silva, L., Aguiar, G., ... Alshehri, M. (2019). Early dropout prediction for programming courses supported by online judges. In International conference on artificial intelligence in education (pp. 67–72). doi: 10.1007/978-3-030-23207-813 [GS Search]
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., Cristea, A. I., Carvalho, L. S. G., Fonseca,S. C., ... Isotani, S. (2020). Using learning analytics in the amazonas: understanding students’ behaviour in introductory programming. British Journal of Educational Technology, 51(4), 955–972. doi: 10.1111/bjet.12953 [GS Search]
Pereira, F. D., Pires, F., Fonseca, S. C., Oliveira, E. H., Carvalho, L. S., Oliveira, D. B., & I, A.(2021). Towards a human-ai hybrid system for categorising programming problems. NewYork, NY, USA: Association for Computing Machinery. doi: 10.1145/3408877.3432422 [GS Search]
Santos, P., Carvalho, L. S. G., Oliveira, E. H. T., & Oliveira, D. B. F. (2019). Classificação de dificuldade de questões de programação com base na inteligibilidade do enunciado. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação– SBIE), 30(1), 1886-1895. doi: 10.5753/cbie.sbie.2019.1886 [GS Search]
Sheard, J., Simon, Carbone, A., Chinn, D., Clear, T., Corney, M., ... Teague, D. (2013). How difficult are exams? a framework for assessing the complexity of introductory programming exams. In Proceedings of the fifteenth australasian computing education conference volume 136 (p. 145—154). AUS: Australian Computer Society, Inc. [GS Search]
Sheard, J., Simon, Carbone, A., Chinn, D., Laakso, M.-J., Clear, T., ... Warburton, G. (2011). Exploring programming assessment instruments: A classification scheme for examination questions. In Proceedings of the seventh international workshop on computing education research (p. 33—38). New York, NY, USA: Association for Computing Machinery. doi: 10.1145/2016911.2016920 [GS Search]
Ullah, F., Wang, J., Farhan, M., Jabbar, S., Wu, Z., & Khalid, S. (2020). Plagiarism detectionin students’ programming assignments based on semantics: multimedia e-learning basedsmart assessment methodology. Multimedia tools and applications, 79(13), 8581–8598. doi: 10.1007/s11042-018-5827-6 [GS Search]
Vargas, A. P., dos Santos, R., Botelho, S. S. C., Tonin, N. A., & Bez, J. L. (2017). Using information technology for personalizing the computer science teaching. In 2017 ieee frontiers ineducation conference (fie)(pp. 1–7). doi: 10.1109/FIE.2017.8190727 [GS Search]
Vargas, A. P., Penna, R., Evandro, Botelho, S. S. C., Tonin, N. A., & Bez, J. L. (2018). A multidimensional elo model for matching learning objects. In 2018 ieee frontiers in educationconference (fie)(p. 1-9). doi: 10.1109/FIE.2018.8658847 [GS Search]
Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259. doi: 10.1016/S0893-6080(05)80023-1 [GS Search]
Zaffalon, F., Vargas, A. P., Souza, R. L., Penna, R., Bez, J. L., Tonin, N. A., & Botelho, S. S. C. (2019). Um estudo comparativo entre dois modelos que estimam a habilidade dos estu-dantes: Elo e teoria de resposta ao item. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE), 30(1), 459–468. doi: 10.5753/c-bie.sbie.2019.459 [GS Search]
Zordan Filho, D. L., de Oliveira, E. H. T., de Carvalho, L. S. G., Pessoa, M., Pereira, F. D., &de Oliveira, D. B. F. (2020). Uma análise orientada a dados para avaliar o impacto dagamificação de um juiz on-line no desempenho de estudantes. In Anais do xxxi simpósio brasileiro de informática na educação (pp. 491–500). doi: 10.5753/cbie.sbie.2020.491 [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
