Evaluating a Collaborative IS to Support Systematic Literature Reviews and Systematic Mapping Studies
DOI:
https://doi.org/10.5753/isys.2023.2919Keywords:
Systematic Literature Review, Systematic Mapping Study, Collaborative System, Software Evaluation, User FeedbackAbstract
This article presents a study to evaluate the quality of the Porifera collaborative tool, which supports Systematic Literature Review (SLR) and Systematic Mapping Study (SMS) processes. The study revealed a high perceived usefulness and ease of use of the tool and identified points for improving the interface, such as user feedback and error prevention. The quantitative and qualitative results suggest that Porifera is a valuable tool for learning and performing collaborative SLR and SMS. Additionally, the article compares Porifera with other related tools, including a discussion on how these other tools were evaluated.
Downloads
References
Al-Zubidy, A. and Carver, J. C. (2019). Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empirical Software Engineering, 24(1):139–169.
Al-Zubidy, A., Carver, J. C., Hale, D. P., and Hassler, E. E. (2017). Vision for SLR tooling infrastructure: Prioritizing value-added requirements. Information and Software Technology, 91:72–81.
Associação Brasileira de Normas Técnicas (2003). NBR ISO/IEC 9126-1: Engenharia de software - Qualidade de produto - Parte 1: Modelo de qualidade.
Bangor, A., Kortum, P., and Miller, J. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating ScaleJUS. Journal of Usability Studies, 4(3):114–123.
Barn, B. S., Raimondi, F., Athappian, L., and Clark, T. (2014). Slrtool: A tool to support collaborative systematic literature reviews. In ICEIS 2014 - Proceedings of the 16th International Conference on Enterprise Information Systems, volume 2, pages 440–447, Lisbon, Portugal. SciTePress.
Basili, V. R. and Caldiera, G. (2000). The Goal Question Metric Paradigm. Encyclopedia of Software Engineering - 2 Volume Set, 2:528–532.
Bigendako, B. and Syriani, E. (2018). Modeling a tool for conducting systematic reviews iteratively. In Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development - MODELSWARD, volume 1, pages 552–559, Madeira, Portugal. SciTePress.
Bowes, D., Hall, T., and Beecham, S. (2012). Slurp: A tool to help large complex systematic literature reviews deliver valid and rigorous results. In Proceedings of the 2nd International Workshop on Evidential Assessment of Software Technologies, EAST ’12, page 33–36, New York, NY, USA. Association for Computing Machinery.
Brooke, J. (1996). SUS: A ’Quick and Dirty’ Usability Guide. InUsability Evaluation InIndustry, chapter 21, page 6. CRC Press, London, UK, 1st edition.
Bustos Navarrete, C., Morales Malverde, M. G., Salcedo Lagos, P., and Díaz Mujica, A. (2018). Buhos: A web-based systematic literature review management software. SoftwareX, 7:360–372.
Campos, T. P. d., Damasceno, E. F., and Valentim, N. M. C. (2022). Proposal and evaluation of a collaborative is to support systematic reviews and mapping studies. In XVIII Brazilian Symposium on Information Systems, SBSI, New York, NY, USA. Association for Computing Machinery.
Corbin, J. and Strauss, A. (2014). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publishing, San Jose, USA, 4th edition.
Davis, F. D. (1993). User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38(3):475–487.
de Brito, J. V. d. C. S. and Ramos, A. S. M. (2019). Limitações dos Modelos de Aceitação da Tecnologia: um Ensaio Sob uma Perspectiva Crítica. Gestão.Org - Revista Eletrônica de Gestão Organizacional, 17(0):210–220.
Fan, M., Shi, S., and Truong, K. N. (2020). Practices and challenges of using think-aloud protocols in industry: An international survey. Journal of Usability Studies, 15(2):85– 102.
Fernández-Sáez, A., Bocco, M., and Romero, F. (2010). SLR-TOOL - A Tool for Performing Systematic Literature Reviews. In Proceedings of the 5th International Conference on Software and Data Technologies, pages 157–166, University of Piraeus, Greece. SciTePress - Science and Technology Publications
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382.
Hassler, E., Carver, J. C., Hale, D., and Al-Zubidy, A. (2016). Identification of SLR tool needs - Results of a community workshop. Information and Software Technology, 70:122–129.
Hernandes, E., Zamboni, A., Fabbri, S., and Di Thommazo, A. (2012). Using GQM and TAM to evaluate StArt – a tool that supports Systematic Review. CLEI Electronic Journal, 15(1):3–3.
Hinderks, A., Jose, F., Mayo, D., Thomaschewski, J., and Escalona, M. J. (2020). An SLR-Tool: Search Process in Practice: To conduct and manage Systematic Literature Review (SLR). In Proceedings - 2020 ACM/IEEE 42nd International Conference on Software Engineering: Companion, ICSE-Companion 2020, pages 81–84, Seoul, South Korea. Institute of Electrical and Electronics Engineers Inc.
ISO (2011). ISO/IEC 25010: Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. Technical report, International Organization for Standardization, Geneva.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical report, Keele University and University of Durham.
Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., and Linkman, S. (2010). Systematic literature reviews in software engineering - A tertiary study.
Kitchenham, B. A., Budgen, D., and Brereton, P. (2015). Evidence-Based Software Engineering and Systematic Reviews. Chapman and CRC Press, Boca Raton, USA, 1st edition.
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., and Wilhelm, R. (2018). Online tools supporting the conduct and reporting of systematic reviews and systematic maps: A case study on CADIMA and review of existing tools. Environmental Evidence, 7(1):1–17.
Marchezan, L., Bolfe, G., Rodrigues, E., Bernardino, M., and Basso, F. P. (2019). Thoth: A Web-based Tool to Support Systematic Reviews. In ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), volume 1, pages 3–8, Porto de Galinhas, Brazil. Institute of Electrical and Electronics Engineers (IEEE).
Marshall, C. (2016). Tool support for systematic reviews in software engineering. PhD thesis, Keele University, Keele, Newcastle, UK.
Marshall, C. and Brereton, P. (2015). Systematic review toolbox: A catalogue of tools to support systematic reviews. In EASE ’15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, volume 1, pages 1–6, Nanjing, China. Association for Computing Machinery.
Marshall, C., Brereton, P., and Kitchenham, B. (2014). Tools to support systematic reviews in software engineering: A feature analysis. In EASE ’14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, pages 1–10, London, UK. Association for Computing Machinery.
Molléri, J. S. and Benitti, F. B. V. (2015). SESRA - A web-based automated tool to support the systematic literature review process. In EASE ’15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, volume 1, pages 1–6, Nanjing, China. Association for Computing Machinery.
Nielsen, J. (2000). Why You Only Need to Test with 5 Users. [link]. Accessed: 2022-09-15.
Nielsen, J. (2012). How Many Test Users in a Usability Study? [link]. Accessed: 2022-09-15.
Okoli, C. and Schabram, K. (2010). A guide to conducting a systematic literature review of information systems research. SSRN Electronic Journal.
Ouzzani, M., Hammady, H., Fedorowicz, Z., and Elmagarmid, A. (2016). Rayyan-a web and mobile app for systematic reviews. Systematic Reviews, 5(1):1–10.
Parsifal Ltd. (2018). About Parsifal. [link]. Accessed: 2021-07-12.
Petersen, K. and Ali, N. B. (2011). Identifying strategies for study selection in systematic reviews and maps. International Symposium on Empirical Software Engineering and Measurement, pages 351–354.
Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008). Systematic mapping studies in software engineering. In Electronic Workshops in Computing. BCS Learning & Development.
Petticrew, M. and Roberts, H. (2005). Systematic Reviews in the Social Sciences: A Practical Guide. Wiley-Blackwell, 1st edition.
Santos, F. M. d. (2012). Análise de conteúdo: a visão de laurence bardin. Revista Eletrônica de Educação, 6(1):383–387.
Silva, R. d. S. e. and Paes, Â. T. (2012). Por Dentro da Estatística: teste de concordância de Kappa. Educ. Contin. Saúde Einstein, 10(4):165–166.
Stefanovic, D., Havzi, S., Nikolic, D., Dakic, D., and Lolic, T. (2021). Analysis of the Tools to Support Systematic Literature Review in Software Engineering. IOP Conference Series: Materials Science and Engineering, 1163(1):012013.
Sutton, A., Clowes, M., Preston, L., and Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. Health Information Libraries Journal, 36(3):202–222.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 iSys - Brazilian Journal of Information Systems
This work is licensed under a Creative Commons Attribution 4.0 International License.
