Assessing the Credibility of Grey Literature: A Study with Brazilian Software Engineering Researchers
DOI:
https://doi.org/10.5753/jserd.2022.1897Keywords:
Grey Literature, Credibility, Empirical Software Engineering, Evidence-Based Software EngineeringAbstract
In recent years, the use and investigations about Grey Literature (GL) increased, in particular, in Software Engineering (SE) research. However, its understanding is still scarce and sometimes controversial, such as interpreting GL types and assessing their credibility. This study aimed to understand the credibility aspects that SE researchers consider in assessing GL and its types. To achieve this goal, we surveyed 53 SE researchers (who answered that they have used GL in our previous investigation), receiving a total of 34 valid responses. Our main findings show that: 1) GL source produced or cited by a renowned source is the main credibility criteria used to assess GL, 2) most of the GL types tend to have a Low to Moderate level of Control and Expertise, 3) there is a positive statistical correlation between the level of Control and Expertise for most GL types, and 4) the different respondent profiles shared similar opinions about the credibility criteria. Our investigation contributes to helping future SE researchers that intend to use GL with more credibility. Additionally, shows the need for future studies to better understand the GL types in SE research.
Downloads
References
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., and Summerbell, M. W. C. (2016a). Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Systematic Reviews, 5(1):164.
Adams, R. J., Smart, P., and Huff, A. S. (2016b). Shades of grey: Guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews, 19(4):432–454.
Baltes, S. and Ralph, P. (2021). Sampling in software engineering research: A critical review and guidelines.
Berg, V., Birkeland, J., Nguyen-Duc, A., Pappas, I. O., and Jaccheri, L. (2018). Software startup engineering: A systematic mapping study. Journal of Systems and Software, 144:255–274.
Bonato, S. (2018). Searching the Grey Literature. Rowman & Littlefield.
Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101.
Coelho, J., Valente, M. T., Milen, L., and Silva, L. L. (2020). Is this GitHub project maintained? measuring the level of maintenance activity of open-source projects. Information and Software Technology, 1:1–35.
Dancey, C. P. and Reidy, J. (2004). Statistics Without Maths for Psychology: Using Spss for Windows. Prentice-Hall, Inc., USA.
Garousi, V., Felderer, M., and Hacaloğlu, T. (2017). Software test maturity assessment and test process improvement: A multivocal literature review. Information and Software Technology, 85:16–42.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2016). The need for multivocal literature reviews in software engineering: Complementing systematic literature reviews with grey literature. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, EASE ’16, pages 26:1–26:6, New York, NY, USA. ACM.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106:101–121.
Hosseinzadeh, S., Rauti, S., Laurén, S., Mäkelä, J.-M., Holvitie, J., Hyrynsalmi, S., and Leppänen, V. (2018). Diversification and obfuscation techniques for software security: A systematic literature review. Information and Software Technology, 104:72–93.
Kamei, F., Wiese, I., Lima, C., Polato, I., Nepomuceno, V., Ferreira, W., Ribeiro, M., Pena, C., Cartaxo, B., Pinto, G., and Soares, S. (2021). Grey literature in software engineering: A critical review. Information and Software Technology, page 106609.
Kamei, F., Wiese, I., Pinto, G., Ribeiro, M., and Soares, S. (2020). On the use of grey literature: A survey with the brazilian software engineering research community. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, SBES 2020, New York, NY, USA. Association for Computing Machinery.
Linåker, J., Sulaman, S., Maiani de Mello, R., and Martin, H. (2015). Guidelines for conducting surveys in software engineering. Technical report, Lund University.
Maro, S., Steghöfer, J.-P., and Staron, M. (2018). Software traceability in the automotive domain: Challenges and solutions. Journal of Systems and Software, 141:85 – 110.
Neto, G. T. G., Santos, W. B., Endo, P. T., and Fagundes, R. A. A. (2019). Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study. In Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’19, pages 1–6.
Oliveira, J. A., Viggiato, M., Pinheiro, D., and Figueiredo, E. (2021). Mining experts from source code analysis: An empirical evaluation. Journal of Software Engineering Research and Development, 9(1):1:1 – 1:16.
Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide, volume 11. Blackwell Publishing Ltd.
Pinto, G., Ferreira, C., Souza, C., Steinmacher, I., and Meirelles, P. (2019). Training software engineers using open-source software: The students’ perspective. In Proceedings of IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET ’19, pages 147–157. Institute of Electrical and Electronics Engineers (IEEE).
Rainer, A. (2017). Using argumentation theory to analyse software practitioners’ feasible evidence, inference and belief. Information and Software Technology, 87:62–80.
Rainer, A. and Williams, A. (2018). Using blog articles in software engineering research: Benefits, challenges and case–survey method. In Proceedings of the 25th Australasian Software Engineering Conference), ASWEC’18, pages 201–209.
Rainer, A. and Williams, A. (2019). Using blog-like documents to investigate software practice: Benefits, challenges, and research directions. Journal of Software: Evolution and Process, 31(11):e2197.
Raulamo-Jurvanen, Päivi, Mäntylä, M., and Garousi, V. (2017). Choosing the right test automation tool: A grey literature review of practitioner sources. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE ’17, pages 21–30. ACM.
Rodríguez-Pérez, G., Robles, G., and González-Barahona, J. M. (2018). Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm. Information and Software Technology, 99:164–176.
Saltan, A. (2019). Do we know how to price saas: A multivocal literature review. In Proceedings of the 2nd ACM SIGSOFT International Workshop on Software-Intensive Business: Start-Ups, Platforms, and Ecosystems, IWSiB 2019, pages 7–12. ACM.
Schöpfel, J. and Prost, H. (2020). How scientific papers mention grey literature: a scientometric study based on scopus data. Collection and Curation.
Soldani, J., Tamburri, D. A., and Heuvel, W.-J. V. D. (2018). The pains and gains of microservices: A systematic grey literature review. Journal of Systems and Software, 146:215–232.
Spencer, D. (2009). Card sorting: Designing usable categories. Rosenfeld Media.
Storey, M.-A., Singer, L., Cleary, B., Filho, F. F., and Zagalsky, A. (2014). The (r) evolution of social media in software engineering. In Proceedings of the on Future of Software Engineering, FOSE ’14. ACM Press.
Storey, M.-A., Zagalsky, A., Filho, F. F., Singer, L., and German, D. M. (2017). How social and communication channels shape and challenge a participatory culture in software development. IEEE Transactions on Software Engineering, 43(2):185–204.
Stray, V. and Moe, N. B. (2020). Understanding coordination in global software engineering: A mixed-methods study on the use of meetings and slack. Journal of Systems and Software, 170:110717.
Tom, E., Aurum, A., and Vidgen, R. (2013). An exploration of technical debt. Journal of Systems and Software, 86(6):1498–1516.
Viera, A. J. and Garrett, J. M. (2005). Understanding interobserver agreement: the kappa statistic. Family Medicine, 37(5):360–363.
Williams, A. (2018). Using reasoning markers to select the more rigorous software practitioners’ online content when searching for grey literature. In Proceedings of the 22Nd International Conference on Evaluation and Assessment in Software Engineering, EASE ’18, pages 46–56. ACM.
Williams, A. and Rainer, A. (2017). Toward the use of blog articles as a source of evidence for software engineering research. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE’17, pages 280–285, New York, NY, USA. ACM.
Williams, A. and Rainer, A. (2019). How do empirical software engineering researchers assess the credibility of practitioner-generated blog posts? In Proceedings of the 23nd International Conference on Evaluation and Assessment in Software Engineering, EASE ’19, pages 211–220. ACM.
Zahedi, M., Rajapakse, R. N., and Babar, M. A. (2020). Mining questions asked about continuous software engineering: A case study of stack overflow. In Li, J., Jaccheri, L., Dingsøyr, T., and Chitchyan, R., editors, EASE ’20: Evaluation and Assessment in Software Engineering, Trondheim, Norway, April 15-17, 2020, pages 41–50. ACM.
Zhang, H., Zhou, X., Huang, X., Huang, H., and Babar, M. A. (2020). An evidence-based inquiry into the use of grey literature in software engineering. In Proceedings of the 42th International Conference on Software Engineering, ICSE ’20.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Fernando Kamei, Igor Wiese, Gustavo Pinto, Waldemar Ferreira, Márcio Ribeiro, Renata Souza, Sérgio Soares
This work is licensed under a Creative Commons Attribution 4.0 International License.
