Cuando los Casos de Prueba No Son Suficientes: Identificación, Evaluación y Justificación de Problemas de Comprensión en Códigos Correctos (PC³)
DOI:
https://doi.org/10.5753/rbie.2023.3552Keywords:
Introducción a la Programación, Problemas de Comprensión, Evaluación Automatizada, autograders, CS1Abstract
Los sistemas de calificación automatizados (autograders) ayudan al proceso de enseñanza en los cursos de introducción a la programación (CS1). Sin embargo, centrarse únicamente en la correctitud puede ofuscar la evaluación de otras características presentes en el código. En este trabajo, investigamos si códigos, considerados correctos por un autograder, fueron desarrollados con características que indiquen posibles comprensiones incorrectas de los conceptos enseñados en CS1. Estas características fueron denominadas Problemas de Comprensión en Códigos Correctos (PC³). Al analizar 2,441 códigos desarrollados por estudiantes de CS1, fue elaborada una lista inicial de 45 PC³. Esta lista fue evaluada por instructores de CS1, lo que resultó en la identificación de los PC³ que más deben abordarse en las clases. Seleccionamos los 15 PC³ más graves para una mayor investigación, incluida una observación semiestructurada en un curso de CS1 y un software de detección automatizado que utiliza análisis de código estático. Los resultados sugieren que los estudiantes desarrollan estos PC³ ya sea debido a una comprensión incompleta de los conceptos enseñados en el curso de CS1 o a una falta de atención al elaborar su código, siendo la correctitud su principal objetivo. Creemos que nuestros resultados pueden contribuir a: (1) el campo de investigación de problemas de comprensión en CS1; (2) promover enfoques alternativos para complementar el uso de autograders en las clases de CS1; y (3) proporcionar conocimientos que puedan servir como base para intervenciones docentes que involucren PC³ en CS1.
Descargas
Citas
Ali, M., Ghosh, S., Rao, P., Dhegaskar, R., Jawort, S., Medler, A., Shi, M., & Dasgupta, S. (2023). Taking stock of concept inventories in computing education: A systematic literature review. 397–415. https://doi.org/10.1145/3568813.3600120. [GS Search]
Almstrum, V. L., Henderson, P. B., Harvey, V., Heeren, C., Marion, W., Riedesel, C., Soh, L.-K., & Tew, A. E. (2006). Concept inventories in computer science for the topic discrete mathematics. Working Group Reports on ITiCSE on Innovation and Technology in Computer Science Education, 132–145. https://doi.org/10.1145/1189215.1189182. [GS Search]
Araujo, A., Filho, D., Oliveira, E., Carvalho, L., Pereira, F., & Oliveira, D. (2021). Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação Anais do Simpósio Brasileiro de Educação em Computação, 123–131. https://doi.org/10.5753/educomp.2021.14478. [GS Search].
Araujo, L. G., Bittencourt, R., & Chavez, C. (2021). Python enhanced error feedback: Uma IDE online de apoio ao processo de ensino-aprendizagem em programação. Anais do Simpósio Brasileiro de Educação em Computação, 326–333. https://doi.org/10.5753/educomp.2021.14500. [GS Search]
Austing, R. H., Barnes, B. H., Bonnette, D. T., Engel, G. L., & Stokes, G. (1979). Curriculum ’78: Recommendations for the Undergraduate Program in Computer Science— a Report of the ACM Curriculum Committee on Computer Science. Commun. ACM, 22(3), 147–166. https://doi.org/10.1145/359080.359083. [GS Search]
Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021). Stop the (autograder) insanity: Regression penalties to deter autograder overreliance. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 1062–1068. https://doi.org/10.1145/3408877.3432430. [GS Search]
Barbosa, A., Costa, E., & Brito, P. (2023). Juízes online são suficientes ou precisamos de um VAR?. Anais do III Simpósio Brasileiro de Educação em Computação, 386–394. https://doi.org/10.5753/educomp.2023.228224. [GS Search]
Becker, B. A., & Fitzpatrick, T. (2019). What do CS1 syllabi reveal about our expectations of introductory programming students?. Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 1011–1017. https://doi.org/10.1145/3287324.3287485. [GS Search]
Becker, B. A., Goslin, K., & Glanville, G. (2018). The effects of enhanced compiler error messages on a syntax error debugging test. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-Janua, 640–645. https://doi.org/10.1145/3159450.3159461. [GS Search]
Bonwell, C., & Eison, J. (1991). Active Learning: Creating Excitement in the Classroom. Wiley. [GS Search]
Bosse, Y., & Gerosa, M. (2015). Reprovações e Trancamentos nas Disciplinas de Introdução à Programação da Universidade de São Paulo: Um Estudo Preliminar. Anais do XXIII Workshop sobre Educação em Computação, 426–435. https://doi.org/10.5753/wei.2015.10259. [GS Search]
Brodley, C. E., Hescott, B. J., Biron, J., Ressing, A., Peiken, M., Maravetz, S., & Mislove, A. (2022). Broadening Participation in Computing via Ubiquitous Combined Majors (CS+X). Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1, 544–550. https://doi.org/10.1145/3478431.3499352. [GS Search]
Caceffo, R., Frank-Bolton, P., Souza, R., & Azevedo, R. (2019). Identifying and validating Java misconceptions toward a CS1 concept inventory. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 23–29. https://doi.org/10.1145/3304221.3319771. [GS Search]
Caceffo, R., Wolfman, S., Booth, K. S., & Azevedo, R. (2016). Developing a computer science concept inventory for introductory programming. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 364–369. https://doi.org/10.1145/2839509.2844559. [GS Search]
Cain, A., & Babar, M. A. (2016). Reflections on applying constructive alignment with formative feedback for teaching introductory programming and software architecture. Proceedings of the 38th International Conference on Software Engineering Companion, 336–345. https://doi.org/10.1145/2889160.2889185. [GS Search]
Cohen, L., Manion, L., & Morrison, K. (2005). Research Methods in Education [GS Search]. Routledge.
Crouch, C. H., & Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970–977. https://doi.org/10.1119/1.1374249. [GS Search]
De Ruvo, G., Tempero, E., Luxton-Reilly, A., Rowe, G. B., & Giacaman, N. (2018). Understanding semantic style by analysing student code. Proceedings of the 20th Australasian Computing Education Conference, 73–82. https://doi.org/10.1145/3160489.3160500. [GS Search]
Dodds, Z., Libeskind-Hadas, R., & Bush, E. (2010). When CS 1 is Biology 1: Cross-disciplinary Collaboration as CS Context. Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, 219–223. https://doi.org/10.1145/1822090.1822152. [GS Search]
Edwards, S. H. (2021). Automated feedback, the next generation: Designing learning experiences. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 610–611. https://doi.org/10.1145/3408877.3437225. [GS Search]
Ellis, M., Shaffer, C. A., & Edwards, S. H. (2019). Approaches for coordinating etextbooks, online programming practice, automated grading, and more into one course. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 126–132. https://doi.org/10.1145/3287324.3287487. [GS Search]
Fisler, K., Krishnamurthi, S., & Siegmund, J. (2016). Modernizing plan-composition studies. Proceedings of the 47th ACM Technical Symposium on Computing Science Education, 211–216. https://doi.org/10.1145/2839509.2844556. [GS Search]
Galvão, L., Fernandes, D., & Gadelha, B. (2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE), 27(1), 140. https://doi.org/10.5753/cbie.sbie.2016.140. [GS Search]
Gama, G., Caceffo, R., Souza, R., Bennati, R., Aparecida, T., Garcia, I., & Azevedo, R. (2018, November). An Antipattern Documentation about Misconceptions related to an Introductory Programming Course in Python (tech. rep. No. IC-18-19). Institute of Computing, University of Campinas. [GS Search]
Guo, P. (2014). Python Is Now the Most Popular Introductory Teaching Language at Top U.S. Universities. Retrieved on 06/26/2023 from [link].
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10), 528–529. https://doi.org/10.1145/367415.367422. [GS Search]
Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K. (2021). Attitudes surrounding an imperfect AI autograder. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445424. [GS Search]
Ihantola, P., & Petersen, A. (2019). Code complexity in introductory programming courses. Proceedings of the 52nd Hawaii International Conference on System Sciences, 7662–7670. [GS Search]
Inside Higher Ed. (2018, November). Autograder issues upset students at berkeley. Retrieved on 06/22/2023 from [link].
Joni, S.-N. A., & Soloway, E. (1986). But My Program Runs! Discourse Rules for Novice Programmers. Journal of Educational Computing Research, 2(1), 95–125. https://doi.org/10.2190/6E5W-AR7C-NX76-HUT2. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2019). How teachers would help students to improve their code. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 119–125. https://doi.org/10.1145/3304221.3319780. [GS Search]
Keuning, H., Heeren, B., & Jeuring, J. (2021). A tutoring system to learn code refactoring. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, 562–568. https://doi.org/10.1145/3408877.3432526. [GS Search]
Kinnunen, P., & Malmi, L. (2006). Why students drop out CS1 course?. Proceedings of the Second International Workshop on Computing Education Research, 97–108. https://doi.org/10.1145/1151588.1151604. [GS Search]
Kluvyer, T. (n.d.). Green Tree Snakes - the missing Python AST docs. Retrieved on 06/05/2023 from [link].
Lazar, J., Feng, J. H., & Hochheiser, H. (2017). Research Methods in Human-Computer Interaction [GS Search]. Morgan Kaufmann.
Lima, M. A. P., Carvalho, L. S. G., Oliveira, E. H. T., Oliveira, D. B. F., & Pereira, F. D. (2021). Uso de atributos de código para classificar a dificuldade de questões de programação em juízes online. Revista Brasileira de Informática na Educação, 29, 1137–1157. https://doi.org/10.5753/rbie.2021.29.0.1137. [GS Search]
Liu, D., & Petersen, A. (2019). Static analyses in Python programming courses. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 666–671. https://doi.org/10.1145/3287324.3287503. [GS Search]
Marwan, S., Lytle, N., Williams, J. J., & Price, T. (2019). The impact of adding textual explanations to next-step hints in a novice programming environment. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, 520–526. https://doi.org/10.1145/3304221.3319759. [GS Search]
Oliveira, E., Keuning, H., & Jeuring, J. (2023). Student code refactoring misconceptions. Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, 19–25. https://doi.org/10.1145/3587102.3588840. [GS Search]
Pereira, F. D., Oliveira, E. H. T., Oliveira, D. B. F., Cristea, A. I., Carvalho, L. S. G., Fonseca, S. C., Toda, A., & Isotani, S. (2020). Using learning analytics in the Amazonas: Understanding students’ behaviour in introductory programming. British Journal of Educational Technology, 51(4), 955–972. https://doi.org/https://doi.org/10.1111/bjet.12953. [GS Search]
Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Oliveira, D. B., Cristea, A. I., & Carvalho, L. S. (2020). Deep learning for early performance prediction of introductory programming students: A comparative and explanatory study. Revista Brasileira de Informática na Educação, 28(0), 723–748. https://doi.org/10.5753/rbie.2020.28.0.723. [GS Search]
Porfirio, A. J., Pereira, R., & Maschio, E. (2021). A-Learn EvId: A Method for Identifying Evidence of Computer Programming Skills Through Automatic Source Code Assessment. Revista Brasileira de Informática na Educação, 29, 692–717. https://doi.org/10.5753/rbie.2021.29.0.692. [GS Search]
Porter, L., Bailey Lee, C., & Simon, B. (2013). Halving fail rates using peer instruction: A study of four computer science courses. Proceedings of the 44th ACM Technical Symposium on Computer Science Education, 177–182. https://doi.org/10.1145/2445196.2445250. [GS Search]
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018). Metacognitive difficulties faced by novice programmers in automated assessment tools. ICER 2018 - Proceedings of the 2018 ACM Conference on International Computing Education Research, 41–50. https://doi.org/10.1145/3230977.3230981. [GS Search]
Qian, Y., & Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1). https://doi.org/10.1145/3077618. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2021). Análise estática de código em conjunto com autograders. Anais Estendidos do I Simpósio Brasileiro de Educação em Computação, 25–26. https://doi.org/10.5753/educomp_estendido.2021.14858. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023a). Misconceptions in Correct Code: rating the severity of undesirable programming behaviors in Python CS1 courses (tech. rep. No. IC-23-01). Institute of Computing, University of Campinas. https://doi.org/10.13140/RG.2.2.28739.89127. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023b). Passar nos casos de teste é suficiente? identificação e análise de problemas de compreensão em códigos corretos. Anais do III Simpósio Brasileiro de Educação em Computação, 119–129. https://doi.org/10.5753/educomp.2023.228346. [GS Search]
Silva, E., Caceffo, R., & Azevedo, R. (2023c). A syllabi analysis of CS1 courses from Brazilian public universities. Brazilian Journal of Computers in Education, 31(1), 407–436. https://doi.org/10.5753/rbie.2023.2870. [GS Search]
Simon, B., Esper, S., Porter, L., & Cutts, Q. (2013). Student experience in a student-centered peer instruction classroom. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, 129–136. https://doi.org/10.1145/2493394.2493407. [GS Search]
Simon, B., Kohanfars, M., Lee, J., Tamayo, K., & Cutts, Q. (2010). Experience report: Peer instruction in introductory computing. Proceedings of the 41st ACM Technical Symposium on Computer Science Education, 341–345. https://doi.org/10.1145/1734263.1734381. [GS Search]
Simon, B., Parris, J., & Spacco, J. (2013). How we teach impacts student learning: Peer instruction vs. lecture in cs0 [GS Search]. Proceeding of the 44th ACM Technical Symposium on Computer Science Education, 41–46. https://doi.org/10.1145/2445196.2445215
Sloan, R. H., Taylor, C., & Warner, R. (2017). Initial Experiences with a CS + Law Introduction to Computer Science (CS 1). Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, 40–45. https://doi.org/10.1145/3059009.3059029. [GS Search]
Soloway, E., & Ehrlich, K. (1984). Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering, SE-10(5), 595–609. https://doi.org/10.1109/TSE.1984.5010283. [GS Search]
Sorva, J., & Vihavainen, A. (2016). Break Statement Considered. ACM Inroads, 7(3), 36–41. https://doi.org/10.1145/2950065. [GS Search]
Tew, A. E., & Guzdial, M. (2011). The FCS1: A Language Independent Assessment of CS1 Knowledge. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, 111–116. https://doi.org/10.1145/1953163.1953200. [GS Search]
Ureel II, L. C., & Wallace, C. (2019). Automated Critique of Early Programming Antipatterns. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 738–744. https://doi.org/10.1145/3287324.3287463. [GS Search]
VanLehn, K. (2006). The behavior of tutoring systems [GS Search]. International journal of artificial intelligence in education, 16(3), 227–265.
Walker, H. M. (2017). ACM RETENTION COMMITTEE Retention of Students in Introductory Computing Courses: Curricular Issues and Approaches. ACM Inroads, 8(4), 14–16. https://doi.org/10.1145/3151936. [GS Search]
Wichmann, B. A., Canning, A., Clutterbuck, D., Winsborrow, L., Ward, N., & Marsh, D. W. R. (1995). Industrial perspective on static analysis. [GS Search]. Software Engineering Journal, 10, 69–75(6).
Archivos adicionales
Published
Cómo citar
Issue
Section
Licencia
Derechos de autor 2023 Eryck Pedro da Silva, Ricardo Caceffo, Rodolfo Azevedo
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
