When Test Cases Are Not Enough: Identification, Assessment, and Rationale of Misconceptions in Correct Code (MC³)
DOI:
https://doi.org/10.5753/rbie.2023.3552Keywords:
Introductory Programming, Misconceptions, Automated Assessment, Autograders, CS1Abstract
Automated grading systems (autograders) assist the process of teaching in introductory programming courses (CS1). However, the sole focus on correctness can obfuscate the assessment of other characteristics present in code. In this work, we investigated if code, deemed correct by an autograder, were developed with characteristics that indicated potential misunderstandings of the concepts taught in CS1. These characteristics were named Misconceptions in Correct Code (MC³). By analyzing 2,441 codes developed by CS1 students, we curated an initial list of 45 MC³. This list was assessed by CS1 instructors, resulting in the identification of MC³ that should be addressed in classes. We selected the 15 most severe MC³ for further investigation, including a semi-structured observation in a CS1 course and an automated detection software using static code analysis. The results suggested that students develop these MC³ either due to an incomplete comprehension of the concepts taught in CS1 course or a lack of attention while elaborating their code, with correctness being their primary goal. We believe our results can contribute to: (1) the research field of misconceptions in CS1; (2) promoting alternative approaches to complement the use of autograders in CS1 classes; and (3) providing insights that can serve as the foundation for teaching interventions involving MC³ in CS1.
Downloads
References
Ali, M., Ghosh, S., Rao, P., Dhegaskar, R., Jawort, S., Medler, A., Shi, M., & Dasgupta, S. (2023). Taking stock of concept inventories in computing education: A systematic literature review. 397–415. https://doi.org/10.1145/3568813.3600120. [GS Search]
Almstrum, V. L., Henderson, P. B., Harvey, V., Heeren, C., Marion, W., Riedesel, C., Soh, L.-K., & Tew, A. E. (2006). Concept inventories in computer science for the topic discrete mathematics. Working Group Reports on ITiCSE on Innovation and Technology in Computer Science Education, 132–145. https://doi.org/10.1145/1189215.1189182. [GS Search]
Araujo, A., Filho, D., Oliveira, E., Carvalho, L., Pereira, F., & Oliveira, D. (2021). Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação Anais do Simpósio Brasileiro de Educação em Computação, 123–131. https://doi.org/10.5753/educomp.2021.14478. [GS Search].
Araujo, L. G., Bittencourt, R., & Chavez, C. (2021). Python enhanced error feedback: Uma IDE online de apoio ao processo de ensino-aprendizagem em programação. Anais do Simpósio Brasileiro de Educação em Computação, 326–333. https://doi.org/10.5753/educomp.2021.14500. [GS Search]
Austing, R. H., Barnes, B. H., Bonnette, D. T., Engel, G. L., & Stokes, G. (1979). Curriculum ’78: Recommendations for the Undergraduate Program in Computer Science— a Report of the ACM Curriculum Committee on Computer Science. Commun. ACM, 22(3), 147–166. https://doi.org/10.1145/359080.359083. [GS Search]
Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021). Stop the (autograder) insanity: Regression penalties to deter autograder overreliance. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 1062–1068. https://doi.org/10.1145/3408877.3432430. [GS Search]
Barbosa, A., Costa, E., & Brito, P. (2023). Juízes online são suficientes ou precisamos de um VAR?. Anais do III Simpósio Brasileiro de Educação em Computação, 386–394. https://doi.org/10.5753/educomp.2023.228224. [GS Search]
Becker, B. A., & Fitzpatrick, T. (2019). What do CS1 syllabi reveal about our expectations of introductory programming students?. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 1011–1017. https://doi.org/10.1145/3287324.3287485. [GS Search]
Becker, B. A., Goslin, K., & Glanville, G. (2018). The effects of enhanced compiler error messages on a syntax error debugging test. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-Janua, 640–645. https://doi.org/10.1145/3159450.3159461. [GS Search]
Bonwell, C., & Eison, J. (1991). Active Learning: Creating Excitement in the Classroom. Wiley. [GS Search]
Bosse, Y., & Gerosa, M. (2015). Reprovações e Trancamentos nas Disciplinas de Introdução à Programação da Universidade de São Paulo: Um Estudo Preliminar. Anais do XXIII Workshop sobre Educação em Computação, 426–435. https://doi.org/10.5753/wei.2015.10259. [GS Search]
Brodley, C. E., Hescott, B. J., Biron, J., Ressing, A., Peiken, M., Maravetz, S., & Mislove, A. (2022). Broadening Participation in Computing via Ubiquitous Combined Majors (CS+X). Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1, 544–550. https://doi.org/10.1145/3478431.3499352. [GS Search]
Caceffo, R., Frank-Bolton, P., Souza, R., & Azevedo, R. (2019). Identifying and validating Java misconceptions toward a CS1 concept inventory. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 23–29. https://doi.org/10.1145/3304221.3319771. [GS Search]
Caceffo, R., Wolfman, S., Booth, K. S., & Azevedo, R. (2016). Developing a computer science concept inventory for introductory programming. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 364–369. https://doi.org/10.1145/2839509.2844559. [GS Search]
Cain, A., & Babar, M. A. (2016). Reflections on applying constructive alignment with formative feedback for teaching introductory programming and software architecture. Proceedings of the 38th International Conference on Software Engineering Companion, 336–345. https://doi.org/10.1145/2889160.2889185. [GS Search]
Cohen, L., Manion, L., & Morrison, K. (2005). Research Methods in Education [GS Search]. Routledge.
Crouch, C. H., & Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970–977. https://doi.org/10.1119/1.1374249. [GS Search]
De Ruvo, G., Tempero, E., Luxton-Reilly, A., Rowe, G. B., & Giacaman, N. (2018). Understanding semantic style by analysing student code. Proceedings of the 20th Australasian Computing Education Conference, 73–82. https://doi.org/10.1145/3160489.3160500. [GS Search]
Dodds, Z., Libeskind-Hadas, R., & Bush, E. (2010). When CS 1 is Biology 1: Cross-disciplinary Collaboration as CS Context. Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, 219–223. https://doi.org/10.1145/1822090.1822152. [GS Search]
Edwards, S. H. (2021). Automated feedback, the next generation: Designing learning experiences. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 610–611. https://doi.org/10.1145/3408877.3437225. [GS Search]
Ellis, M., Shaffer, C. A., & Edwards, S. H. (2019). Approaches for coordinating etextbooks, online programming practice, automated grading, and more into one course. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 126–132. https://doi.org/10.1145/3287324.3287487. [GS Search]
Fisler, K., Krishnamurthi, S., & Siegmund, J. (2016). Modernizing plan-composition studies. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 211–216. https://doi.org/10.1145/2839509.2844556. [GS Search]
Galvão, L., Fernandes, D., & Gadelha, B. (2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE), 27(1), 140. https://doi.org/10.5753/cbie.sbie.2016.140. [GS Search]
Gama, G., Caceffo, R., Souza, R., Bennati, R., Aparecida, T., Garcia, I., & Azevedo, R. (2018, November). An Antipattern Documentation about Misconceptions related to an Introductory Programming Course in Python (tech. rep. No. IC-18-19). Institute of Computing, University of Campinas. [GS Search]
Guo, P. (2014). Python Is Now the Most Popular Introductory Teaching Language at Top U.S. Universities. Retrieved on 06/26/2023 from [link].
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10), 528–529. https://doi.org/10.1145/367415.367422. [GS Search]
Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K. (2021). Attitudes surrounding an imperfect AI autograder. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445424. [GS Search]
Ihantola, P., & Petersen, A. (2019). Code complexity in introductory programming courses. Proceedings of the 52nd Hawaii International Conference on System Sciences, 7662–7670. [GS Search]
Inside Higher Ed. (2018, November). Autograder issues upset students at berkeley. Retrieved on 06/22/2023 from [link].
Joni, S.-N. A., & Soloway, E. (1986). But My Program Runs! Discourse Rules for Novice Programmers. Journal of Educational Computing Research, 2(1), 95–125. https://doi.org/10.2190/6E5W-AR7C-NX76-HUT2. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2019). How teachers would help students to improve their code. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 119–125. https://doi.org/10.1145/3304221.3319780. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2021). A tutoring system to learn code refactoring. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 562–568. https://doi.org/10.1145/3408877.3432526. [GS Search]
Kinnunen, P., & Malmi, L. (2006). Why students drop out CS1 course?. Proceedings of the Second International Workshop on Computing Education Research, 97–108. https://doi.org/10.1145/1151588.1151604. [GS Search]
Kluvyer, T. (n.d.). Green Tree Snakes - the missing Python AST docs. Retrieved on 06/05/2023 from [link].
Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Research Methods in Human-Computer Interaction [GS Search]. Morgan Kaufmann.
Lima, M. A. P., Carvalho, L. S. G., Oliveira, E. H. T., Oliveira, D. B. F., & Pereira, F. D. (2021). Uso de atributos de código para classificar a dificuldade de questões de programação em juízes online. Revista Brasileira de Informática na Educação, 29, 1137–1157. https://doi.org/10.5753/rbie.2021.29.0.1137. [GS Search]
Liu, D., & Petersen, A. (2019). Static analyses in Python programming courses. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 666–671. https://doi.org/10.1145/3287324.3287503. [GS Search]
Marwan, S., Lytle, N., Williams, J. J., & Price, T. (2019). The impact of adding textual explanations to next-step hints in a novice programming environment. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 520–526. https://doi.org/10.1145/3304221.3319759. [GS Search]
Oliveira, E., Keuning, H., & Jeuring, J. (2023). Student code refactoring misconceptions. Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, 19–25. https://doi.org/10.1145/3587102.3588840. [GS Search]
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., Cristea, A. I., Carvalho, L. S. G., Fonseca, S. C., Toda, A., & Isotani, S. (2020). Using learning analytics in the Amazonas: Understanding students’ behaviour in introductory programming. British Journal of Educational Technology, 51(4), 955–972. https://doi.org/https://doi.org/10.1111/bjet.12953. [GS Search]
Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., & Carvalho, L. S. (2020). Deep learning for early performance prediction of introductory programming students: A comparative and explanatory study. Revista Brasileira de Informática na Educação, 28(0), 723–748. https://doi.org/10.5753/rbie.2020.28.0.723. [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2021). A-Learn EvId: A Method for Identifying Evidence of Computer Programming Skills Through Automatic Source Code Assessment. Revista Brasileira de Informática na Educação, 29, 692–717. https://doi.org/10.5753/rbie.2021.29.0.692. [GS Search]
Porter, L., Bailey Lee, C., & Simon, B. (2013). Halving fail rates using peer instruction: A study of four computer science courses. Proceedings of the 44th ACM Technical Symposium on Computer Science Education, 177–182. https://doi.org/10.1145/2445196.2445250. [GS Search]
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018). Metacognitive difficulties faced by novice programmers in automated assessment tools. ICER 2018 - Proceedings of the 2018 ACM Conference on International Computing Education Research, 41–50. https://doi.org/10.1145/3230977.3230981. [GS Search]
Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1). https://doi.org/10.1145/3077618. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2021). Análise estática de código em conjunto com autograders. Anais Estendidos do I Simpósio Brasileiro de Educação em Computação, 25–26. https://doi.org/10.5753/educomp_estendido.2021.14858. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023a). Misconceptions in Correct Code: rating the severity of undesirable programming behaviors in Python CS1 courses (tech. rep. No. IC-23-01). Institute of Computing, University of Campinas. https://doi.org/10.13140/RG.2.2.28739.89127. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023b). Passar nos casos de teste é suficiente? identificação e análise de problemas de compreensão em códigos corretos. Anais do III Simpósio Brasileiro de Educação em Computação, 119–129. https://doi.org/10.5753/educomp.2023.228346. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023c). A syllabi analysis of CS1 courses from Brazilian public universities. Brazilian Journal of Computers in Education, 31(1), 407–436. https://doi.org/10.5753/rbie.2023.2870. [GS Search]
Simon, B., Esper, S., Porter, L., & Cutts, Q. (2013). Student experience in a student-centered peer instruction classroom. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, 129–136. https://doi.org/10.1145/2493394.2493407. [GS Search]
Simon, B., Kohanfars, M., Lee, J., Tamayo, K., & Cutts, Q. (2010). Experience report: Peer instruction in introductory computing. Proceedings of the 41st ACM Technical Symposium on Computer Science Education, 341–345. https://doi.org/10.1145/1734263.1734381. [GS Search]
Simon, B., Parris, J., & Spacco, J. (2013). How we teach impacts student learning: Peer instruction vs. lecture in cs0 [GS Search]. Proceeding of the 44th ACM Technical Symposium on Computer Science Education, 41–46. https://doi.org/10.1145/2445196.2445215
Sloan, R. H., Taylor, C., & Warner, R. (2017). Initial Experiences with a CS + Law Introduction to Computer Science (CS 1). Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 40–45. https://doi.org/10.1145/3059009.3059029. [GS Search]
Soloway, E., & Ehrlich, K. (1984). Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering, SE-10(5), 595–609. https://doi.org/10.1109/TSE.1984.5010283. [GS Search]
Sorva, J., & Vihavainen, A. (2016). Break Statement Considered. ACM Inroads, 7(3), 36–41. https://doi.org/10.1145/2950065. [GS Search]
Tew, A. E., & Guzdial, M. (2011). The FCS1: A Language Independent Assessment of CS1 Knowledge. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, 111–116. https://doi.org/10.1145/1953163.1953200. [GS Search]
Ureel II, L. C., & Wallace, C. (2019). Automated Critique of Early Programming Antipatterns. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 738–744. https://doi.org/10.1145/3287324.3287463. [GS Search]
VanLehn, K. (2006). The behavior of tutoring systems [GS Search]. International journal of artificial intelligence in education, 16(3), 227–265.
Walker, H. M. (2017). ACM RETENTION COMMITTEE Retention of Students in Introductory Computing Courses: Curricular Issues and Approaches. ACM Inroads, 8(4), 14–16. https://doi.org/10.1145/3151936. [GS Search]
Wichmann, B. A., Canning, A., Clutterbuck, D., Winsborrow, L., Ward, N., & Marsh, D. W. R. (1995). Industrial perspective on static analysis. [GS Search]. Software Engineering Journal, 10, 69–75(6).
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Eryck Pedro da Silva, Ricardo Caceffo, Rodolfo Azevedo
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
