A systematic mapping about the peer assessment of competencies of an individual in teams of software projects
DOI:
https://doi.org/10.5753/isys.2020.797Keywords:
Performance assessment, peer assessment, software project teamAbstract
The success of a software project relies on individual’s competences of a team. For this, it is necessary that each individual is evaluated so that it is possible to determine the competences that it possesses and how they can be performed in a software project. This article aims to perform a systematic mapping of the literature to identify how peer evaluation is being used in evaluating software project teams and what competencies are evaluated in this context. Four research questions were defined and used to guide a search in nine mechanisms of academic search during the period of September, 2018. From 363 documents found, after the inclusion and exclusion criteria, 23 articles were selected for further reading. Based on 23 articles selected, it is possible to identify that there are different types, methods, strategies and tools used in the peer evaluation that support this process, but the competences are not very explored in this point, as well as there are opportunities of research in the professional area, since most articles report only academic experiences.
Downloads
Download data is not yet available.
References
Aguiar, G., Kemczinski, A., and Gasparini, I. (2016). The automated formation of corporate groups for software projects: A systematic mapping. Proceedings of the XII Brazilian Symposium on Information Systems on Brazilian Symposium on Information Systems: Information Systems in the Cloud Computing Era - Volume 1, SBSI 2016,pages 80:605–80:612, Porto Alegre, Brazil, Brazil. Brazilian Computer Society.
Anewalt, K. (2005). Using peer review as a vehicle for communication skill development and active learning. J. Comput. Sci. Coll., 21(2):148–155.
Barry, B. E., Drnevich, V. P., Irfanoglu, A., and Bullock, D. (2011). Summary of developments in the civil engineering capstone course at purdue university. Journal of Professional Issues in Engineering Education & Practice, 138(1):95–98.
Bayot, P. C. P. (2011). Avaliação por competência no mundo globalizado.
Bloom, B. S., Hostings, J., and Madaus, G. F. (1975). Evaluación del aprendizaje.
Brandão, H. P., Zimmer, M. V., Pereira, C. G., Marques, F., Costa, H. V., Carbone, P. P., and Almada, V. F. (2008). Gestão de desempenho por competências: integrando a gestão por competências, o balanced scorecard e a avaliação 360 graus. Revista de Administração Pública, 42(5):875–898.
Brodie, L. and Gibbings, P. (2009). Comparison of pbl assessment rubrics. Research in Engineering Education Symposium.
Brodie, L. and Kist, A. (2011). Mapping learning outcomes and assignment tasks for spider activities. International Journal of Emerging Technologies in Learning (iJET), 6(2011):25–32.
Buchinger, D., de Siqueira Cavalcanti, G. A., and da Silva Hounsell, M. (2014). Mecanis- mos de busca acadêmica: uma análise quantitativa. Revista Brasileira de Computação Aplicada, 6(1):108–120.
Caballero, J. A., Palomo, M., Dodero, J. M., Rodriguez, G., and Ibarra, M. S. (2014). Integrating external evidences of skill assessment in virtual learning environments.
Cannon-Bowers, J. A. and Salas, E. (1998). Team performance and training in complex environments: Recent findings from applied research. Current Directions in Psycho- logical Science, 7(3):83–87.
Capello, C. (2013). Avaliação e o novo cenário da educação. 3(1):4–10.
Cassidy, S. (2006). Developing employability skills: peer assessment in higher education. Education + Training, 48:508–517.
Cebrián-Robles, D. (2016). Corubric.
Chatenier, E. d., Verstegen, J. A. A. M., Biemans, H. J. A., Mulder, M., and Omta, O. S. W. F. (2010). Identification of competencies for professionals in open innovation teams. R&D Management, 40(3):271–280.
Coens, T. and Jenkins, M. (2002). Abolishing performance appraisals: Why they backfire and what to do instead.
Deist, F. D. L. and Winterton, J. (2005). What is competence? Human Resource Deve- lopment International, 8(1):27–46.
Derntl, M. (2006). A generic view on web-supported peer-evaluation. pages 1809–1816.
Dochy, F., Segers, M., and Sluijsmans, D. (1999). The use of self-, peer and co-assessment in higher education: A review. In Studies in Higher Education, 24(3):331–350.
Dornan, T., Lee, C., Stopford, A., Hosie, L., Maredia, N., and Rector, A. (2005). Rapid application design of an electronic clinical skills portfolio for undergraduate medical students. Computer methods and programs in biomedicine, 78(1):25–33.
Duzer, E. V. and McMartin, F. (2000). Methods to improve the validity and sensitivity of a self/peer assessment instrument. 43(2):153–158.
Fagerholm, F. and Vihavainen, A. (2013). Peer assessment in experiential learning assessing tacit and explicit skills in agile software engineering capstone projects. In Frontiers in Education Conference, 2013 IEEE, pages 1723–1729. IEEE.
Farrell, V., Farrell, G., Kindler, P., Ravalli, G., and Hall, D. (2013). Capstone project online assessment tool without the paper work. In Proceedings of the 18th ACM con- ference on Innovation and technology in computer science education, pages 201–206. ACM.
Fernández-Araóz, C. (2014). 21st-century talent spotting. 92:46–54.
Figl, K., Bauer, C., and Mangler, J. (2006). Online versus face-to-face peer team reviews. In Frontiers in Education Conference, 36th Annual, pages 7–12. IEEE.
Fleury, M. T. L. and Fleury, A. (2001). Construindo o conceito de competência. Revista de administração contemporânea, 5:183–196.
Fox, B. I., Umphress, D. A., and Hollingsworth, J. C. (2017). Development and delivery of an interdisciplinary course in mobile health (mhealth). Currents in Pharmacy Teaching and Learning, 9(4):585–594.
Freeman, M. and McKenzie, J. (2002). Spark, a confidential web–based template for self and peer assessment of student teamwork: benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5):551–569.
Glaser, R. (1963). Instructional technology and the measurement of learing outcomes: Some questions. 18(8):519.
Goldfinch, J. and Raeside, R. (1990). evelopment of a peer assessment technique for obtaining individual marks on a group project. 15(3):210–231.
Herbert, N. (2018). Reflections on 17 years of ict capstone project coordination: Effective strategies for managing clients, teams and assessment. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, pages 215–220. ACM.
Isson, J. P. and Harriot, J. S. (2016). People analytics in the era of big data: Changing the way you attract, acquire, develop, and retain talent. John Wiley & Sons.
Johns-Boast, L. and Flint, S. (2013). Simulating industry: An innovative software engineering capstone design course. In Frontiers in Education Conference, 2013 IEEE, pages 1782–1788. IEEE.
Kane, J. S. and Lawler, E. E. (1978). Methods of peer assessment. Psychological Bulletin, 85(3):555–586.
Kemavuthanon, K. (2017). Integrated e-project collaborative management system: Empirical study for problem-based learning project. In Information Technology and Elec- trical Engineering (ICITEE), 2017 9th International Conference on, pages 1–5. IEEE.
Kitchenham, A. B., Dyba, T., and Jorgensen, M. (2004). Evidence-based software engine- ering. Proceedings of the 26th International Conference on Software Engineering, pa- ges 273–281. http://dl.acm.org/citation.cfm?id=998675.999432.
Klock, A. C. T. (2018). Mapeamentos e revisões sistemáticos da literatura: um guia teórico e prático. Cadernos de Informática, 10(1):01–09.
Leme, R. (2006). Avaliação de desempenho com foco em competência. Qualitymark Editora Ltda.
Marshall, L., Pieterse, V., Thompson, L., and Venter, D. M. (2016). Exploration of participation in student software engineering teams. ACM Transactions on Computing Education (TOCE), 16(2):5.
Mathieu, J., Maynard, M. T., Rapp, T., and Gilson, L. (2008). Team effectiveness 1997- 2007: A review of recent advancements and a glimpse into the future. Journal of management, 34(3):410–476.
McHenry, W. K. (2015). Selecting teams: The potential of “the grid”. Education and Information Technologies, 20(1):183–199.
McKinney, D. and Denton, L. F. (2006). Developing collaborative skills early in the cs curriculum in a laboratory environment. In ACM SIGCSE Bulletin, volume 38, pages 138–142. ACM.
Murray, J. (2013). Likert data: What to use, parametric or non-parametric? International Journal of Business and Social Science, 4(11):258–264.
Ohland, M. W., Bullard, L., Felder, R., Finelli, C., Layton, R., Loughry, M., and Sch- mucker, D. (2005). The comprehensive assessment of team-member effectiveness. software created as part of NSF DUE-ASA Award, 11(4).
Passos, M. F. D. (2014). Elaboração e validação de escala de diferencial semântico para avaliação de personalidade. pages 01–134.
Pérez-Torregrosa, A.-B., Torres-Hernández, N., and Ibáñez Cubillas, P. (2018). Evalu- ación con tecnologías de la competencia trabajo en equipo: Un enfoque dialógico y participativo. 09:28–35.
Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008). Systematic mapping studies in software engineering. 8:68–77.
Rethans, J.-J., Norcini, J., Baron-Maldonado, M., Blackmore, D., Jolly, B., LaDuca, T., Lew, S., Page, G., and Southgate, L. (2002). The relationship between competence and performance: implications for assessing practice performance. Medical education, 36(10):901–909.
Rosen, C. C. (1996). Individual assessment of group projects in software engineering: A facilitated peer assessment approach. In Software Engineering Education, 1996. Proceedings., Ninth Conference on, pages 68–77. IEEE.
Rovamo, L., Nurmi, E., Mattila, M.-M., Suominen, P., and Silvennoinen, M. (2015). Effect of a simulation-based workshop on multidisplinary teamwork of newborn emergencies: an intervention study. BMC research notes, 8(1):671.
Ryan, G. J., Marshall, L. L., Porter, K., and Jia, H. (2007). Peer, professor and self- evaluation of class participation. 8(1):49–61.
Sain, S. and Wilde, S. (2014). Review of soft skills within knowledge management. pages 7–55.
Sáiz, M. S. I. e. a. (2010). Evalcomix en moodle: Un medio para favorecer la participación de los estudiantes en la e-evaluación.
Samath, S., Udalagama, D., Kurukulasooriya, H., Premarathne, D., and Thelijjagoda, S. (2017). Collabcrew—an intelligent tool for dynamic task allocation within a software development team. In Software, Knowledge, Information Management and Applications (SKIMA), 2017 11th International Conference on, pages 1–9. IEEE.
Scheuren, F. (2004). What is a survey?
Simmonds, J., Samary, M. M., Tomic, M., Madrid, F., and Escobar, C. (2018). Software engineering for millennials, by millennials. arXiv preprint arXiv:1804.03518.
Sims-Knight, J. E., Upchurch, R. L., Powers, T., Haden, S., and Topciu, R. (2002). Teams in software engineering education. In Frontiers in Education, 2002. FIE 2002. 32nd Annual, volume 3, pages S3G–S3G. IEEE.
Sitthiworachart, J. and Joy, M. (2003). Web-based peer assessment in learning computer programming. pages 180–184.
Smith III, H. H. and Smarkusky, D. L. (2005). Competency matrices for peer assessment of individuals in team projects. Proceedings of the 6th conference on Information technology education, pages 155–162.
Stiggins, R. J. (1987). Design and development of performance assessments. Educational measurement: Issues and practice, 6(3):33–42.
Topping, K. J., Smith, E. F., Swanson, I., and Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & evaluation in higher education, 25(2):149–169.
Traverso-Ribón, I., Ruíz-Rube, I., Dodero, J. M., and Palomo-Duarte, M. (2013). Open data framework for sustainable assessment in software forges. In Proceedings of the 3rd International Conference on Web Intelligence, Mining and Semantics, page 20. ACM.
Tursunbayeva, A., Di Lauro, S., and Pagliari, C. (2018). People analytics—a scoping review of conceptual boundaries and value propositions. International Journal of Information Management, 43:224–247.
Upchurch, R. L. and Sims-Knight, J. E. (2004). Teaming assessment: is there a connection between process and product? In Frontiers in Education, 2004. FIE 2004. 34th Annual, pages S3C–21. IEEE.
Walsh, B. (2017). Rewriting the rules for the digital age 2017: Deloitte global human capital trends.
Willey, K. and Gardner, A. (2009). Investigating the potential of self and peer assessment to develop learning oriented assessment tasks.
Young, P., Yip, V., and Lenin, R. (2012). Evaluation of issue-tracker’s effectiveness for measuring individual performance on group projects. In Proceedings of the 50th Annual Southeast Regional Conference, pages 89–94. ACM.
Zarifian, P. (2001). Objetivo competência: por uma nova lógica; tradução Maria Helena. 7(4).
Anewalt, K. (2005). Using peer review as a vehicle for communication skill development and active learning. J. Comput. Sci. Coll., 21(2):148–155.
Barry, B. E., Drnevich, V. P., Irfanoglu, A., and Bullock, D. (2011). Summary of developments in the civil engineering capstone course at purdue university. Journal of Professional Issues in Engineering Education & Practice, 138(1):95–98.
Bayot, P. C. P. (2011). Avaliação por competência no mundo globalizado.
Bloom, B. S., Hostings, J., and Madaus, G. F. (1975). Evaluación del aprendizaje.
Brandão, H. P., Zimmer, M. V., Pereira, C. G., Marques, F., Costa, H. V., Carbone, P. P., and Almada, V. F. (2008). Gestão de desempenho por competências: integrando a gestão por competências, o balanced scorecard e a avaliação 360 graus. Revista de Administração Pública, 42(5):875–898.
Brodie, L. and Gibbings, P. (2009). Comparison of pbl assessment rubrics. Research in Engineering Education Symposium.
Brodie, L. and Kist, A. (2011). Mapping learning outcomes and assignment tasks for spider activities. International Journal of Emerging Technologies in Learning (iJET), 6(2011):25–32.
Buchinger, D., de Siqueira Cavalcanti, G. A., and da Silva Hounsell, M. (2014). Mecanis- mos de busca acadêmica: uma análise quantitativa. Revista Brasileira de Computação Aplicada, 6(1):108–120.
Caballero, J. A., Palomo, M., Dodero, J. M., Rodriguez, G., and Ibarra, M. S. (2014). Integrating external evidences of skill assessment in virtual learning environments.
Cannon-Bowers, J. A. and Salas, E. (1998). Team performance and training in complex environments: Recent findings from applied research. Current Directions in Psycho- logical Science, 7(3):83–87.
Capello, C. (2013). Avaliação e o novo cenário da educação. 3(1):4–10.
Cassidy, S. (2006). Developing employability skills: peer assessment in higher education. Education + Training, 48:508–517.
Cebrián-Robles, D. (2016). Corubric.
Chatenier, E. d., Verstegen, J. A. A. M., Biemans, H. J. A., Mulder, M., and Omta, O. S. W. F. (2010). Identification of competencies for professionals in open innovation teams. R&D Management, 40(3):271–280.
Coens, T. and Jenkins, M. (2002). Abolishing performance appraisals: Why they backfire and what to do instead.
Deist, F. D. L. and Winterton, J. (2005). What is competence? Human Resource Deve- lopment International, 8(1):27–46.
Derntl, M. (2006). A generic view on web-supported peer-evaluation. pages 1809–1816.
Dochy, F., Segers, M., and Sluijsmans, D. (1999). The use of self-, peer and co-assessment in higher education: A review. In Studies in Higher Education, 24(3):331–350.
Dornan, T., Lee, C., Stopford, A., Hosie, L., Maredia, N., and Rector, A. (2005). Rapid application design of an electronic clinical skills portfolio for undergraduate medical students. Computer methods and programs in biomedicine, 78(1):25–33.
Duzer, E. V. and McMartin, F. (2000). Methods to improve the validity and sensitivity of a self/peer assessment instrument. 43(2):153–158.
Fagerholm, F. and Vihavainen, A. (2013). Peer assessment in experiential learning assessing tacit and explicit skills in agile software engineering capstone projects. In Frontiers in Education Conference, 2013 IEEE, pages 1723–1729. IEEE.
Farrell, V., Farrell, G., Kindler, P., Ravalli, G., and Hall, D. (2013). Capstone project online assessment tool without the paper work. In Proceedings of the 18th ACM con- ference on Innovation and technology in computer science education, pages 201–206. ACM.
Fernández-Araóz, C. (2014). 21st-century talent spotting. 92:46–54.
Figl, K., Bauer, C., and Mangler, J. (2006). Online versus face-to-face peer team reviews. In Frontiers in Education Conference, 36th Annual, pages 7–12. IEEE.
Fleury, M. T. L. and Fleury, A. (2001). Construindo o conceito de competência. Revista de administração contemporânea, 5:183–196.
Fox, B. I., Umphress, D. A., and Hollingsworth, J. C. (2017). Development and delivery of an interdisciplinary course in mobile health (mhealth). Currents in Pharmacy Teaching and Learning, 9(4):585–594.
Freeman, M. and McKenzie, J. (2002). Spark, a confidential web–based template for self and peer assessment of student teamwork: benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5):551–569.
Glaser, R. (1963). Instructional technology and the measurement of learing outcomes: Some questions. 18(8):519.
Goldfinch, J. and Raeside, R. (1990). evelopment of a peer assessment technique for obtaining individual marks on a group project. 15(3):210–231.
Herbert, N. (2018). Reflections on 17 years of ict capstone project coordination: Effective strategies for managing clients, teams and assessment. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, pages 215–220. ACM.
Isson, J. P. and Harriot, J. S. (2016). People analytics in the era of big data: Changing the way you attract, acquire, develop, and retain talent. John Wiley & Sons.
Johns-Boast, L. and Flint, S. (2013). Simulating industry: An innovative software engineering capstone design course. In Frontiers in Education Conference, 2013 IEEE, pages 1782–1788. IEEE.
Kane, J. S. and Lawler, E. E. (1978). Methods of peer assessment. Psychological Bulletin, 85(3):555–586.
Kemavuthanon, K. (2017). Integrated e-project collaborative management system: Empirical study for problem-based learning project. In Information Technology and Elec- trical Engineering (ICITEE), 2017 9th International Conference on, pages 1–5. IEEE.
Kitchenham, A. B., Dyba, T., and Jorgensen, M. (2004). Evidence-based software engine- ering. Proceedings of the 26th International Conference on Software Engineering, pa- ges 273–281. http://dl.acm.org/citation.cfm?id=998675.999432.
Klock, A. C. T. (2018). Mapeamentos e revisões sistemáticos da literatura: um guia teórico e prático. Cadernos de Informática, 10(1):01–09.
Leme, R. (2006). Avaliação de desempenho com foco em competência. Qualitymark Editora Ltda.
Marshall, L., Pieterse, V., Thompson, L., and Venter, D. M. (2016). Exploration of participation in student software engineering teams. ACM Transactions on Computing Education (TOCE), 16(2):5.
Mathieu, J., Maynard, M. T., Rapp, T., and Gilson, L. (2008). Team effectiveness 1997- 2007: A review of recent advancements and a glimpse into the future. Journal of management, 34(3):410–476.
McHenry, W. K. (2015). Selecting teams: The potential of “the grid”. Education and Information Technologies, 20(1):183–199.
McKinney, D. and Denton, L. F. (2006). Developing collaborative skills early in the cs curriculum in a laboratory environment. In ACM SIGCSE Bulletin, volume 38, pages 138–142. ACM.
Murray, J. (2013). Likert data: What to use, parametric or non-parametric? International Journal of Business and Social Science, 4(11):258–264.
Ohland, M. W., Bullard, L., Felder, R., Finelli, C., Layton, R., Loughry, M., and Sch- mucker, D. (2005). The comprehensive assessment of team-member effectiveness. software created as part of NSF DUE-ASA Award, 11(4).
Passos, M. F. D. (2014). Elaboração e validação de escala de diferencial semântico para avaliação de personalidade. pages 01–134.
Pérez-Torregrosa, A.-B., Torres-Hernández, N., and Ibáñez Cubillas, P. (2018). Evalu- ación con tecnologías de la competencia trabajo en equipo: Un enfoque dialógico y participativo. 09:28–35.
Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008). Systematic mapping studies in software engineering. 8:68–77.
Rethans, J.-J., Norcini, J., Baron-Maldonado, M., Blackmore, D., Jolly, B., LaDuca, T., Lew, S., Page, G., and Southgate, L. (2002). The relationship between competence and performance: implications for assessing practice performance. Medical education, 36(10):901–909.
Rosen, C. C. (1996). Individual assessment of group projects in software engineering: A facilitated peer assessment approach. In Software Engineering Education, 1996. Proceedings., Ninth Conference on, pages 68–77. IEEE.
Rovamo, L., Nurmi, E., Mattila, M.-M., Suominen, P., and Silvennoinen, M. (2015). Effect of a simulation-based workshop on multidisplinary teamwork of newborn emergencies: an intervention study. BMC research notes, 8(1):671.
Ryan, G. J., Marshall, L. L., Porter, K., and Jia, H. (2007). Peer, professor and self- evaluation of class participation. 8(1):49–61.
Sain, S. and Wilde, S. (2014). Review of soft skills within knowledge management. pages 7–55.
Sáiz, M. S. I. e. a. (2010). Evalcomix en moodle: Un medio para favorecer la participación de los estudiantes en la e-evaluación.
Samath, S., Udalagama, D., Kurukulasooriya, H., Premarathne, D., and Thelijjagoda, S. (2017). Collabcrew—an intelligent tool for dynamic task allocation within a software development team. In Software, Knowledge, Information Management and Applications (SKIMA), 2017 11th International Conference on, pages 1–9. IEEE.
Scheuren, F. (2004). What is a survey?
Simmonds, J., Samary, M. M., Tomic, M., Madrid, F., and Escobar, C. (2018). Software engineering for millennials, by millennials. arXiv preprint arXiv:1804.03518.
Sims-Knight, J. E., Upchurch, R. L., Powers, T., Haden, S., and Topciu, R. (2002). Teams in software engineering education. In Frontiers in Education, 2002. FIE 2002. 32nd Annual, volume 3, pages S3G–S3G. IEEE.
Sitthiworachart, J. and Joy, M. (2003). Web-based peer assessment in learning computer programming. pages 180–184.
Smith III, H. H. and Smarkusky, D. L. (2005). Competency matrices for peer assessment of individuals in team projects. Proceedings of the 6th conference on Information technology education, pages 155–162.
Stiggins, R. J. (1987). Design and development of performance assessments. Educational measurement: Issues and practice, 6(3):33–42.
Topping, K. J., Smith, E. F., Swanson, I., and Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & evaluation in higher education, 25(2):149–169.
Traverso-Ribón, I., Ruíz-Rube, I., Dodero, J. M., and Palomo-Duarte, M. (2013). Open data framework for sustainable assessment in software forges. In Proceedings of the 3rd International Conference on Web Intelligence, Mining and Semantics, page 20. ACM.
Tursunbayeva, A., Di Lauro, S., and Pagliari, C. (2018). People analytics—a scoping review of conceptual boundaries and value propositions. International Journal of Information Management, 43:224–247.
Upchurch, R. L. and Sims-Knight, J. E. (2004). Teaming assessment: is there a connection between process and product? In Frontiers in Education, 2004. FIE 2004. 34th Annual, pages S3C–21. IEEE.
Walsh, B. (2017). Rewriting the rules for the digital age 2017: Deloitte global human capital trends.
Willey, K. and Gardner, A. (2009). Investigating the potential of self and peer assessment to develop learning oriented assessment tasks.
Young, P., Yip, V., and Lenin, R. (2012). Evaluation of issue-tracker’s effectiveness for measuring individual performance on group projects. In Proceedings of the 50th Annual Southeast Regional Conference, pages 89–94. ACM.
Zarifian, P. (2001). Objetivo competência: por uma nova lógica; tradução Maria Helena. 7(4).
Downloads
Published
2020-06-17
How to Cite
Santos, T. da R., Pazini, E., & Kemczinski, A. (2020). A systematic mapping about the peer assessment of competencies of an individual in teams of software projects. ISys - Brazilian Journal of Information Systems, 13(3), 98–131. https://doi.org/10.5753/isys.2020.797
Issue
Section
Surveys
