On the Adoption of Empirical Methods and Systematic Reviews in the Brazilian Symposium on Human Factors in Computing Systems

Mariela I. Cortés; Sávio Freire; Lucas Vieira Alves; Matheus Lima Chagas; Marx Haron Barbosa; Adson Damasceno; Andressa Ferreira; Eliakim Gama; José Paulo Rodrigues Moraes

doi:10.5753/jis.2021.981

Authors

Mariela I. Cortés State University of Ceará http://orcid.org/0000-0002-6731-8123
Sávio Freire Federal Institute of Ceará https://orcid.org/0000-0002-3989-9442
Lucas Vieira Alves State University of Ceará https://orcid.org/0000-0003-4320-3329
Matheus Lima Chagas State University of Ceará https://orcid.org/0000-0002-1263-8090
Marx Haron Barbosa State University of Ceará
Adson Damasceno State University of Ceará
Andressa Ferreira State University of Ceará
Eliakim Gama State University of Ceará https://orcid.org/0000-0003-3880-7144
José Paulo Rodrigues Moraes State University of Ceará

DOI:

https://doi.org/10.5753/jis.2021.981

Keywords:

empirical evaluation, quality assessment, human-computer interaction research

Abstract

Context: Empirical studies (ES) and systematic reviews (SR) play an essential role in the Human-Computer Interaction (HCI) field as its focus is on evaluating the end-user and usability of software solutions and synthesizing the evidence found by the HCI community. Even though the adoption of empirical evaluation techniques and SR has gained popularity in recent years, the consistent use of a methodology is still maturing. Goal: This study aims to provide a qualitative and quantitative assessment of the current status of ES and SR presented in the research papers published at the proceedings of the Brazilian Symposium on Human Factors in Computing Systems (IHC Symposium). Method: We conduct an empirical study on the papers over the 18 editions in the IHC Symposium to answer four research questions. Our study proposes a protocol to identify and assess ES and SR reported in the papers published at the IHC Symposium. Results: From the sample of 259 studies, we find 131 ES and SR (~51%). We have characterized and categorized the ES into case studies, experiments, and surveys. Further, we found evidence that these studies' quantity and quality have been increased over the IHC Symposium editions, and almost half of these studies give detailed information making possible their replication. Conclusion: We hope that each study's characterization can support the conduction of new ES and SR by the HCI Brazilian community, producing more reliable results and reducing or eliminating biases.

Downloads

Download data is not yet available.

References

Barbosa, D. M., Gadelha, R., Alencar, T., Neves, B., Yeltsin, I., Gomes, T., and Cortés, M. I. (2017). An analysis of the empirical software engineering over the last 10 editions of brazilian software engineering symposium. In Proceedings of the 31st Brazilian Symposium on Software Engineering, SBES 2017, Fortaleza, CE, Brazil, September 20-22, 2017, pages 44–53.

Basili, V. R. (1996). The role of experimentation in software engineering: Past, current, and future. In Proceedings of the 18th International Conference on Software Engineering (ICSE), pages 442–449. IEEE Computer Society.

Damasceno, A., Ferreira, A., Gama, E., Moraes, J. P. R., Alves, L. V., Barbosa, M. H., Chagas, M. L., Freire, E. S. S., and Cortés, M. I. (2019). A landscape of the adoption of empirical evaluations in the brazilian symposium on human factors in computing systems. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, IHC ’19, New York, NY, USA. Association for Computing Machinery.

Fink, A. (2003). The Survey Handbook. SAGE Publications.

Gergle, D. and Tan, D. (2014). Experimental Research in HCI. Olson J., Kellogg W. (eds) Ways of Knowing in HCI. Springer, New York, NY.

Juzgado, N. J. and Gómez, O. S. (2010). Replication of software engineering experiments. In Meyer, B. and Nordio, M., editors, Empirical Software Engineering and Verification - International Summer Schools, LASER 2008-2010, Elba Island, Italy, Revised Tutorial Lectures, volume 7007 of Lecture Notes in Computer Science, pages 60–88. Springer.

Juzgado, N. J. and Moreno, A. M. (2001). Basics of software engineering experimentation. Kluwer.

Karlström, D. and Runeson, P. (2006). Integrating agile software development into stage-gate managed product development. Empirical Software Engineering, 11(2):203–225.

Keele, S. (2007). Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn.

Kitchenham, B., Madeyski, L., and Brereton, P. (2019). Problems with statistical practice in human-centric software engineering experiments. In Proceedings of the Evaluation and Assessment on Software Engineering, EASE ’19, page 134–143, New York, NY, USA. Association for Computing Machinery.

Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., and Linkman, S. (2009a). Systematic literature reviews in software engineering - a systematic literature review. Information and Software Technology, 51(1):7–15.

Kitchenham, B. A., Brereton, O. P., Budgen, D., and Li, Z. (2009b). An evaluation of quality checklist proposals: A participant-observer case study. In Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering, EASE’09, pages 55–64, Swindon, UK. BCS Learning & Development Ltd.

Kitchenham, B. A., Budgen, D., and Brereton, P. (2015). Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall/CRC.

Kitchenham, B. A., Dyba, T., and Jorgensen, M. (2004). Evidence-based software engineering. In Proceedings of the 26th international conference on software engineering, pages 273–281. IEEE Computer Society.

Kitchenham, B. A., Mendes, E., and Travassos, G. H. (2007). Cross versus within-company cost estimation studies: A systematic review. IEEE Transactions on Software Engineering, 33(5):316–329.

Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., and Rosenberg, J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721–734.

Lazar, J., Feng, J. H., and Hochheiser, H. (2017). Research Methods in Human-Computer Interaction. Morgan Kaufmann, Cambridge, MA, 2 edition.

Levenberg, K. (1944). A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 2(2):164–168.

Linåker, J., Sulaman, S. M., Maiani de Mello, R., and Höst, M. (2015). Guidelines for conducting surveys in software engineering. http://portal.research.lu.se/portal/files/6062997/5463412.pdf. Accessed 10 December 2016.

MacKenzie, I. S. (2013). Human-Computer Interaction: An Empirical Research Perspective. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1st edition.

Malhotra, R. (2015). Empirical Research in Software Engineering: Concepts, Analysis, and Applications. Chapman & Hall/CRC.

McKnight, P. E. and Najab, J. (2010). Mann-whitney u test. The Corsini encyclopedia of psychology, pages 1–1.

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., and Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1:0021.

Olson, J. S. and Kellogg, W. A. (2014). Ways of Knowing in HCI. Springer Publishing Company, Incorporated.

Runeson, P., Host, M., Rainer, A., and Regnell, B. (2012). Case Study Research in Software Engineering: Guidelines and Examples. Wiley Publishing.

Sammut, C. and Webb, G. I. (2011). Encyclopedia of machine learning. Springer Science & Business Media.

Serrano, J. F., Acuña, S. T., and Macías, J. A. (2014). A review of quantitative empirical approaches in humancomputer interaction. In Proceedings of the XV International Conference on Human Computer Interaction, pages 1–8.

Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S., and Elmqvist, N. (2016). Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Boston, 6 edition.

Silveira Neto, P. A. D. M., Gomes, J. S., De Almeida, E. S., Leite, J. C., Batista, T. V., and Leite, L. (2013). 25 years of software engineering in brazil: Beyond an insider’s view. J. Syst. Softw., 86(4):872–889.

Sjoberg, D. I. K., Dyba, T., and Jorgensen, M. (2007). The future of empirical methods in software engineering research. In Proceedings of the Future of Software Engineering (FOSE), pages 358–378. IEEE Computer Society.

Valverde, R. (2011). Principles of Human Computer Interaction Design.

Vargha, A. and Delaney, H. D. (2000). A critique and improvement of the cl common language effect size statistics of mcgraw and wong. Journal of Educational and Behavioral Statistics, 25(2):101–132.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in software engineering. Springer Science & Business Media.

Wohlin, C. and Wesslen, A. (1998). Understanding software defect detection in the personal software process. In Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257), pages 49–58.

Zannier, C., Melnik, G., and Maurer, F. (2006). On the success of empirical studies in the international conference on software engineering. In Proceedings of the 28th International Conference on Software Engineering, ICSE ’06, pages 341–350, New York, NY, USA. ACM.