Quando Casos de Teste Não São Suficientes: Identificação, Avaliação e Justificativas de Problemas de Compreensão em Códigos Corretos (PC³)
DOI:
https://doi.org/10.5753/rbie.2023.3552Keywords:
Programação Introdutória, Avaliação Automática, problemas de compreensão, autograders, CS1Abstract
Sistemas de avaliação automática de código (autograders) auxiliam no processo de ensino em disciplinas de introdução à programação (CS1). Contudo, o foco exclusivo na correção pode ofuscar a avaliação de outras características presentes no código. Neste trabalho, investigamos se códigos, considerados corretos por um autograder, foram desenvolvidos com características que indicavam possíveis compreensões incorretas dos conceitos ensinados em CS1. Essas características foram denominadas Problemas de Compreensão em Códigos Corretos (PC³). Ao analisar 2.441 códigos desenvolvidos por alunos de CS1, foi elaborada uma lista inicial de 45 PC³. Essa lista foi avaliada por instrutores de CS1, resultando na identificação dos PC³ que deveriam ser corrigidos em sala de aula. Os 15 MC³ mais graves foram selecionados para uma investigação maior, composta por uma observação semiestruturada em um curso de CS1 e um software de detecção automática usando análise estática de código. Os resultados obtidos mostraram que os alunos desenvolvem esses PC³ ou em decorrência de uma compreensão incompleta dos conceitos ensinados na disciplina CS1 ou por falta de atenção na elaboração de seu código, porque o aluno só estava focando na corretude. Acreditamos que nossos resultados podem contribuir para: (1) a área de pesquisa de problemas de compreensão em CS1; (2) promover abordagens alternativas para complementar o uso de autograders em CS1; e (3) fornecer uma base para o desenvolvimento de intervenções de ensino envolvendo os PC³ em CS1.
Downloads
Referências
Ali, M., Ghosh, S., Rao, P., Dhegaskar, R., Jawort, S., Medler, A., Shi, M., & Dasgupta, S. (2023). Taking stock of concept inventories in computing education: A systematic literature review. 397–415. https://doi.org/10.1145/3568813.3600120. [GS Search]
Almstrum, V. L., Henderson, P. B., Harvey, V., Heeren, C., Marion, W., Riedesel, C., Soh, L.-K., & Tew, A. E. (2006). Concept inventories in computer science for the topic discrete mathematics. Working Group Reports on ITiCSE on Innovation and Technology in Computer Science Education, 132–145. https://doi.org/10.1145/1189215.1189182. [GS Search]
Araujo, A., Filho, D., Oliveira, E., Carvalho, L., Pereira, F., & Oliveira, D. (2021). Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação Anais do Simpósio Brasileiro de Educação em Computação, 123–131. https://doi.org/10.5753/educomp.2021.14478. [GS Search].
Araujo, L. G., Bittencourt, R., & Chavez, C. (2021). Python enhanced error feedback: Uma IDE online de apoio ao processo de ensino-aprendizagem em programação. Anais do Simpósio Brasileiro de Educação em Computação, 326–333. https://doi.org/10.5753/educomp.2021.14500. [GS Search]
Austing, R. H., Barnes, B. H., Bonnette, D. T., Engel, G. L., & Stokes, G. (1979). Curriculum ’78: Recommendations for the Undergraduate Program in Computer Science— a Report of the ACM Curriculum Committee on Computer Science. Commun. ACM, 22(3), 147–166. https://doi.org/10.1145/359080.359083. [GS Search]
Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021). Stop the (autograder) insanity: Regression penalties to deter autograder overreliance. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 1062–1068. https://doi.org/10.1145/3408877.3432430. [GS Search]
Barbosa, A., Costa, E., & Brito, P. (2023). Juízes online são suficientes ou precisamos de um VAR?. Anais do III Simpósio Brasileiro de Educação em Computação, 386–394. https://doi.org/10.5753/educomp.2023.228224. [GS Search]
Becker, B. A., & Fitzpatrick, T. (2019). What do CS1 syllabi reveal about our expectations of introductory programming students?. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 1011–1017. https://doi.org/10.1145/3287324.3287485. [GS Search]
Becker, B. A., Goslin, K., & Glanville, G. (2018). The effects of enhanced compiler error messages on a syntax error debugging test. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-Janua, 640–645. https://doi.org/10.1145/3159450.3159461. [GS Search]
Bonwell, C., & Eison, J. (1991). Active Learning: Creating Excitement in the Classroom. Wiley. [GS Search]
Bosse, Y., & Gerosa, M. (2015). Reprovações e Trancamentos nas Disciplinas de Introdução à Programação da Universidade de São Paulo: Um Estudo Preliminar. Anais do XXIII Workshop sobre Educação em Computação, 426–435. https://doi.org/10.5753/wei.2015.10259. [GS Search]
Brodley, C. E., Hescott, B. J., Biron, J., Ressing, A., Peiken, M., Maravetz, S., & Mislove, A. (2022). Broadening Participation in Computing via Ubiquitous Combined Majors (CS+X). Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1, 544–550. https://doi.org/10.1145/3478431.3499352. [GS Search]
Caceffo, R., Frank-Bolton, P., Souza, R., & Azevedo, R. (2019). Identifying and validating Java misconceptions toward a CS1 concept inventory. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 23–29. https://doi.org/10.1145/3304221.3319771. [GS Search]
Caceffo, R., Wolfman, S., Booth, K. S., & Azevedo, R. (2016). Developing a computer science concept inventory for introductory programming. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 364–369. https://doi.org/10.1145/2839509.2844559. [GS Search]
Cain, A., & Babar, M. A. (2016). Reflections on applying constructive alignment with formative feedback for teaching introductory programming and software architecture. Proceedings of the 38th International Conference on Software Engineering Companion, 336–345. https://doi.org/10.1145/2889160.2889185. [GS Search]
Cohen, L., Manion, L., & Morrison, K. (2005). Research Methods in Education [GS Search]. Routledge.
Crouch, C. H., & Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970–977. https://doi.org/10.1119/1.1374249. [GS Search]
De Ruvo, G., Tempero, E., Luxton-Reilly, A., Rowe, G. B., & Giacaman, N. (2018). Understanding semantic style by analysing student code. Proceedings of the 20th Australasian Computing Education Conference, 73–82. https://doi.org/10.1145/3160489.3160500. [GS Search]
Dodds, Z., Libeskind-Hadas, R., & Bush, E. (2010). When CS 1 is Biology 1: Cross-disciplinary Collaboration as CS Context. Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, 219–223. https://doi.org/10.1145/1822090.1822152. [GS Search]
Edwards, S. H. (2021). Automated feedback, the next generation: Designing learning experiences. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 610–611. https://doi.org/10.1145/3408877.3437225. [GS Search]
Ellis, M., Shaffer, C. A., & Edwards, S. H. (2019). Approaches for coordinating etextbooks, online programming practice, automated grading, and more into one course. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 126–132. https://doi.org/10.1145/3287324.3287487. [GS Search]
Fisler, K., Krishnamurthi, S., & Siegmund, J. (2016). Modernizing plan-composition studies. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 211–216. https://doi.org/10.1145/2839509.2844556. [GS Search]
Galvão, L., Fernandes, D., & Gadelha, B. (2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE), 27(1), 140. https://doi.org/10.5753/cbie.sbie.2016.140. [GS Search]
Gama, G., Caceffo, R., Souza, R., Bennati, R., Aparecida, T., Garcia, I., & Azevedo, R. (2018, November). An Antipattern Documentation about Misconceptions related to an Introductory Programming Course in Python (tech. rep. No. IC-18-19). Institute of Computing, University of Campinas. [GS Search]
Guo, P. (2014). Python Is Now the Most Popular Introductory Teaching Language at Top U.S. Universities. Retrieved on 06/26/2023 from [link].
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10), 528–529. https://doi.org/10.1145/367415.367422. [GS Search]
Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K. (2021). Attitudes surrounding an imperfect AI autograder. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445424. [GS Search]
Ihantola, P., & Petersen, A. (2019). Code complexity in introductory programming courses. Proceedings of the 52nd Hawaii International Conference on System Sciences, 7662–7670. [GS Search]
Inside Higher Ed. (2018, November). Autograder issues upset students at berkeley. Retrieved on 06/22/2023 from [link].
Joni, S.-N. A., & Soloway, E. (1986). But My Program Runs! Discourse Rules for Novice Programmers. Journal of Educational Computing Research, 2(1), 95–125. https://doi.org/10.2190/6E5W-AR7C-NX76-HUT2. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2019). How teachers would help students to improve their code. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 119–125. https://doi.org/10.1145/3304221.3319780. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2021). A tutoring system to learn code refactoring. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 562–568. https://doi.org/10.1145/3408877.3432526. [GS Search]
Kinnunen, P., & Malmi, L. (2006). Why students drop out CS1 course?. Proceedings of the Second International Workshop on Computing Education Research, 97–108. https://doi.org/10.1145/1151588.1151604. [GS Search]
Kluvyer, T. (n.d.). Green Tree Snakes - the missing Python AST docs. Retrieved on 06/05/2023 from [link].
Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Research Methods in Human-Computer Interaction [GS Search]. Morgan Kaufmann.
Lima, M. A. P., Carvalho, L. S. G., Oliveira, E. H. T., Oliveira, D. B. F., & Pereira, F. D. (2021). Uso de atributos de código para classificar a dificuldade de questões de programação em juízes online. Revista Brasileira de Informática na Educação, 29, 1137–1157. https://doi.org/10.5753/rbie.2021.29.0.1137. [GS Search]
Liu, D., & Petersen, A. (2019). Static analyses in Python programming courses. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 666–671. https://doi.org/10.1145/3287324.3287503. [GS Search]
Marwan, S., Lytle, N., Williams, J. J., & Price, T. (2019). The impact of adding textual explanations to next-step hints in a novice programming environment. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 520–526. https://doi.org/10.1145/3304221.3319759. [GS Search]
Oliveira, E., Keuning, H., & Jeuring, J. (2023). Student code refactoring misconceptions. Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, 19–25. https://doi.org/10.1145/3587102.3588840. [GS Search]
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., Cristea, A. I., Carvalho, L. S. G., Fonseca, S. C., Toda, A., & Isotani, S. (2020). Using learning analytics in the Amazonas: Understanding students’ behaviour in introductory programming. British Journal of Educational Technology, 51(4), 955–972. https://doi.org/https://doi.org/10.1111/bjet.12953. [GS Search]
Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., & Carvalho, L. S. (2020). Deep learning for early performance prediction of introductory programming students: A comparative and explanatory study. Revista Brasileira de Informática na Educação, 28(0), 723–748. https://doi.org/10.5753/rbie.2020.28.0.723. [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2021). A-Learn EvId: A Method for Identifying Evidence of Computer Programming Skills Through Automatic Source Code Assessment. Revista Brasileira de Informática na Educação, 29, 692–717. https://doi.org/10.5753/rbie.2021.29.0.692. [GS Search]
Porter, L., Bailey Lee, C., & Simon, B. (2013). Halving fail rates using peer instruction: A study of four computer science courses. Proceedings of the 44th ACM Technical Symposium on Computer Science Education, 177–182. https://doi.org/10.1145/2445196.2445250. [GS Search]
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018). Metacognitive difficulties faced by novice programmers in automated assessment tools. ICER 2018 - Proceedings of the 2018 ACM Conference on International Computing Education Research, 41–50. https://doi.org/10.1145/3230977.3230981. [GS Search]
Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1). https://doi.org/10.1145/3077618. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2021). Análise estática de código em conjunto com autograders. Anais Estendidos do I Simpósio Brasileiro de Educação em Computação, 25–26. https://doi.org/10.5753/educomp_estendido.2021.14858. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023a). Misconceptions in Correct Code: rating the severity of undesirable programming behaviors in Python CS1 courses (tech. rep. No. IC-23-01). Institute of Computing, University of Campinas. https://doi.org/10.13140/RG.2.2.28739.89127. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023b). Passar nos casos de teste é suficiente? identificação e análise de problemas de compreensão em códigos corretos. Anais do III Simpósio Brasileiro de Educação em Computação, 119–129. https://doi.org/10.5753/educomp.2023.228346. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023c). A syllabi analysis of CS1 courses from Brazilian public universities. Brazilian Journal of Computers in Education, 31(1), 407–436. https://doi.org/10.5753/rbie.2023.2870. [GS Search]
Simon, B., Esper, S., Porter, L., & Cutts, Q. (2013). Student experience in a student-centered peer instruction classroom. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, 129–136. https://doi.org/10.1145/2493394.2493407. [GS Search]
Simon, B., Kohanfars, M., Lee, J., Tamayo, K., & Cutts, Q. (2010). Experience report: Peer instruction in introductory computing. Proceedings of the 41st ACM Technical Symposium on Computer Science Education, 341–345. https://doi.org/10.1145/1734263.1734381. [GS Search]
Simon, B., Parris, J., & Spacco, J. (2013). How we teach impacts student learning: Peer instruction vs. lecture in cs0 [GS Search]. Proceeding of the 44th ACM Technical Symposium on Computer Science Education, 41–46. https://doi.org/10.1145/2445196.2445215
Sloan, R. H., Taylor, C., & Warner, R. (2017). Initial Experiences with a CS + Law Introduction to Computer Science (CS 1). Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 40–45. https://doi.org/10.1145/3059009.3059029. [GS Search]
Soloway, E., & Ehrlich, K. (1984). Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering, SE-10(5), 595–609. https://doi.org/10.1109/TSE.1984.5010283. [GS Search]
Sorva, J., & Vihavainen, A. (2016). Break Statement Considered. ACM Inroads, 7(3), 36–41. https://doi.org/10.1145/2950065. [GS Search]
Tew, A. E., & Guzdial, M. (2011). The FCS1: A Language Independent Assessment of CS1 Knowledge. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, 111–116. https://doi.org/10.1145/1953163.1953200. [GS Search]
Ureel II, L. C., & Wallace, C. (2019). Automated Critique of Early Programming Antipatterns. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 738–744. https://doi.org/10.1145/3287324.3287463. [GS Search]
VanLehn, K. (2006). The behavior of tutoring systems [GS Search]. International journal of artificial intelligence in education, 16(3), 227–265.
Walker, H. M. (2017). ACM RETENTION COMMITTEE Retention of Students in Introductory Computing Courses: Curricular Issues and Approaches. ACM Inroads, 8(4), 14–16. https://doi.org/10.1145/3151936. [GS Search]
Wichmann, B. A., Canning, A., Clutterbuck, D., Winsborrow, L., Ward, N., & Marsh, D. W. R. (1995). Industrial perspective on static analysis. [GS Search]. Software Engineering Journal, 10, 69–75(6).
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2023 Eryck Pedro da Silva, Ricardo Caceffo, Rodolfo Azevedo
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
