A-Learn EvId: A Method for Identifying Evidence of Computer Programming Skills Through Automatic Source Code Assessment
DOI:
https://doi.org/10.5753/rbie.2021.29.0.692Keywords:
Computer programming, Automatic assessment, Skills identificationAbstract
Contextualized in the teaching of computer programming in Computing courses, this research investigates aspects and strategies for automatic source code assessment. Continuous on-time assessment of source codes produced by students is a challenging task for teachers. The literature presents different methods for automatic assessment of source code, mostly focusing on technical aspects, such as functional correctness assessment and error detection. This paper presents the A-Learn EvId method, having as the main characteristic its focus on the assessment of high-level skills instead of technical aspects. Automatically assessing high-level skills gives insights into the thought process students used to elaborate their responses, contributing to quality and timely feedback generation. The method is characterized by three fundamental steps: (1) inserting students' source code as input data; (2) identifying evidence of skills through automatic strategies; and (3) representing identified skills through a student model. The following contributions are highlighted: updating the state of the art on the topic; a set of 37 skills identifiable through 9 automatic source code assessment strategies; construction of datasets totaling 8651 source codes.
Downloads
Referências
Ahmed, U. Z., Kumar, P., Karkare, A., Kar, P., & Gulwani, S. (2018). Compilation error repair: For the student programs, from the student programs. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering Education and Training (pp. 78– 87). New York, NY, USA: ACM. doi: 10.1145/3183377.3183383[GS Search]
Cui, B., Li, J., Guo, T., Wang, J., & Ma, D. (2010, Oct). Code comparison system based on abstract syntax tree. In 2010 3rd IEEE International Conference on Broadband Network and Multimedia Technology (IC-BNMT) (p. 668-673). doi: 10.1109/ICBNMT.2010.5705174[GS Search]
Edmison, B., & Edwards, S. H. (2019). Experiences using heat maps to help students find their bugs: Problems and solutions. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 260–266). New York, NY, USA: ACM. doi: 10.1145/3287324.3287474[GS Search]
Gerdt, P., & Sajaniemi, J. (2006, June). A web-based service for the automatic detection of roles of variables. SIGCSE Bull., 38(3), 178–182. doi: 10.1145/1140123.1140172 [GS Search]
Hettiarachchi, E., Huertas, M., & Mor, E. (2013, 12). Skill and knowledge e-assessment: A review of the state of the art. IN3 Working Paper Series. doi: 10.7238/in3wps.v0i0.1958 [GS Search]
Hettiarachchi, E., Huertas, M., & Mor, E. (2015, 01). E-assessment system for skill and knowledge assessment in computer engineering education. International Journal of Engineering Education, 31, 529–540. [GS Search]
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (pp. 86–93). New York, NY, USA: ACM. doi: 10.1145/1930464.1930480 [GS Search]
Jackson, D., & Usher, M. (1997). Grading student programs using assyst. In Proceedings of the Twenty-eighth SIGCSE Technical Symposium on Computer Science Education (pp. 335–339). New York, NY, USA: ACM. doi: 10.1145/268084.268210 [GS Search]
Kautzmann, T., & Jaques, P. (2020). Modelo de identificação de unidades de conhecimento de programação em processo de aplicação durante a codificação. In Anais do XXXI Simpósio Brasileiro de Informática na Educação (pp. 982–991). Porto Alegre, RS, Brasil: SBC. doi: 10.5753/cbie.sbie.2020.982 [GS Search]
Koh, K. H., Nickerson, H., Basawapatna, A., & Repenning, A. (2014). Early validation of computational thinking pattern analysis. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (pp. 213–218). New York, NY, USA: ACM. doi: 10.1145/2591708.2591724 [GS Search]
Liang, Y., Liu, Q., Xu, J., & Wang, D. (2009, Dec). The recent development of automated programming assessment. In 2009 International Conference on Computational Intelligence and Software Engineering (p. 1-5). doi: 10.1109/CISE.2009.5365307 [GS Search]
Maschio, E. (2013). Modelagem do processo de aquisição de conhecimento apoiado por ambientes inteligentes. Tese de doutorado, Programa de Pós-Graduação em Informática, Setor de Ciências Exatas, Universidade Federal do Paraná (UFPR). [GS Search]
Morris, D. S. (2003, Nov). Automatic grading of student’s programming assignments: an interactive process and suite of programs. In 33rd Annual Frontiers in Education, 2003. FIE 2003. (Vol. 3, p. S3F-1). doi: 10.1109/FIE.2003.1265998 [GS Search]
Neapolitan, R. E. (2003). Learning bayesian networks. Upper Saddle River, NJ, USA: PrenticeHall, Inc. [GS Search]
Pimentel, M., & Carvalho, F. d. S. P. (2020, 05). Princípios da educação online: para sua aula não ficar massiva nem maçante! SBC Horizontes. SBC Horizontes, maio 2020. ISSN 2175-9235. Disponível em: [Link]. [GS Search]
Porfirio, A. J. (2020). Identifying evidences of computer programming skills through automatic source code evaluation. Phd thesis, Programa de Pós-Graduação em Informática, Setor de Ciências Exatas, Universidade Federal do Paraná (UFPR). [GS Search]
Porfirio, A. J., Maschio, E., & Direne, A. (2016). Modelagem genérica de aprendizes com Ênfase em erros na aquisição de habilidades em programação de computadores. Anais dos Workshops do Congresso Brasileiro de Informática na Educação. doi: 10.5753/cbie.wcbie.2016.1198 [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2017). Atualização do modelo do aprendiz de programação de computadores com o uso de parser ast. Anais dos Workshops do Congresso Brasileiro de Informática na Educação, 6(1), 1121. doi: 10.5753/cbie.wcbie.2017.1121 [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2018). Inferência de conhecimento a partir da detecção automática de evidências no domínio da programação de computadores. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE), 29(1), 1553. doi: 10.5753/cbie.sbie.2018.1553 [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2020). Identifying evidences of computer programming skills through automatic source code evaluation. In Anais dos Workshops do IX Congresso Brasileiro de Informática na Educação (pp. 01–10). Porto Alegre, RS, Brasil: SBC. doi: 10.5753/cbie.wcbie.2020.01 [GS Search]
Rahman, K. A., & Nordin, M. J. (2007). A review on the static analysis approach in the automated programming assessment systems. In Proceedings of National Conference on Programming 07. [GS Search]
Rajala, T., Kaila, E., Lindén, R., Kurvinen, E., Lokkila, E., Laakso, M.-J., & Salakoski, T. (2016). Automatically assessed electronic exams in programming courses. In Proceedings of the Australasian Computer Science Week Multiconference (pp. 11:1–11:8). New York, NY, USA: ACM. doi: 10.1145/2843043.2843062 [GS Search]
Souza, D. M., Felizardo, K. R., & Barbosa, E. F. (2016, April). A systematic literature review of assessment tools for programming assignments. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET) (p. 147-156). doi: 10.1109/CSEET.2016.48 [GS Search]
Turing, A. M. (1936). On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, 2(42), 230–265. Retrieved from [link] [GS Search]
Ullah, Z., Lajis, A., Jamjoom, M., Altalhi, A., Al-Ghamdi, A., & Saleem, F. (2018). The effect of automatic assessment on novice programming: Strengths and limitations of existing systems. Computer Applications in Engineering Education, 26(6), 2328–2341. [GS Search]
VanPatten, B., & Williams, J. (2015). Theories in second language acquisition: An introduction (Second ed.). Routledge. [GS Search]
Wilcox, T. R., Davis, A. M., & Tindall, M. H. (1976, November). The design and implementation of a table driven, interactive diagnostic programming system. Commun. ACM, 19(11), 609–616. doi: 10.1145/360363.360367 [GS Search]
Wu, J. (2011, 12). Improving the writing of research papers: IMRAD and beyond. Landscape Ecology, 26. doi: 10.1007/s10980-011-9674-3 [GS Search]
Yamashita, K., Sugiyama, T., Kogure, S., Noguchi, Y., Konishi, T., & Itoh, Y. (2017). An educational support system based on automatic impasse detection in programming exercises. In Proceedings of the 25th International Conference on Computers in Education, ICCE 2017 - Main Conference Proceedings (p. 288-295). Retrieved from [Link]. [GS Search]
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
