Journal of Software Engineering Research and Development

The Influence of Search Engine Ranking Quality on the Performance of Programmers

Adriano M. Rocha, Carlos E. C. Dantas, Marcelo de Almeida Maia — Mon, 16 Jun 2025 00:00:00 +0000

Software development is an activity characterized by the continuous search for information by programmers. General-purpose search engines, such as Google, Bing, and Yahoo, are widely used by programmers to find solutions to their problems. However, the best solutions are not always among the initial pages of search results that are ranked according to the search engine's algorithms. This study aims to investigate the influence that the order of the pages returned by the ranking of search engines exerts on programmers' performance when performing programming tasks. We designed an empirical within-subject study with programmers to understand and evaluate their performance when solving programming tasks using a ranked list of pages returned by the Google search engine and artificially modified with two different ranking quality levels (Higher Quality Ranking and Lower Quality Ranking). Moreover, for each entry in the ranking of pages, the most frequent methods mentioned on the respective page were listed in the ranking visualization. Analysis of participants' recorded videos was conducted through a mixed-methods research approach to provide insights into the results. We found that programmers spent approximately 8 minutes longer resolving tasks associated with a Lower Quality Ranking, spending more time on irrelevant pages compared to relevant ones, due to efforts to fix problematic code or new searches for another page. The addition of a list of frequent methods in the ranking visualization could help programmers to skip irrelevant pages and reduce time wastage. The ranking quality influences the programmers' performance during the development of programming tasks. Therefore, we suggest the development of filters aimed at improving the quality of results delivered by search engines. Moreover, the results may encourage the adaptation of this study for other approaches that require information foraging, such as chatting with LLMs.

Test case prioritization methods: A systematic literature review

Icaro Prado Fernandes, Luiz Eduardo Galvão Martins — Mon, 16 Jun 2025 00:00:00 +0000

Software testing is a process that assists in achieving software quality. However, software testing used to be a repetitive activity that takes a long time to be performed. Several methods for test case prioritization have been created with the purpose to reduce the time of software testing activity without compromising its quality. The main idea of this review is to examine and classify test case prioritization methods based on the research questions created. In order to achieve the outlined objective, seventy-one primary studies that propose, analyze, or compare test case prioritization methods were used as a basis for the analysis. Sixty-eight of them were found by search string and four studies were manually included by the author. The main methods of each group were mapped, analyzed and described. Despite the existence of many methods for prioritizing test cases, none of them were designed especially to be applied in agile environments, therefore there are opportunities for improvement in this topic.

Feature selection in an interactive search-based PLA design approach

Wed, 28 May 2025 00:00:00 +0000

The Product Line Architecture (PLA) is one of the most important artifacts of a Software Product Line (SPL). PLA design can be formulated as an interactive optimization problem with many conflicting factors. Incorporating Decision Makers’ (DM) preferences during the search process may help the algorithms find more adequate solutions for their profiles. Interactive approaches allow the DM to evaluate solutions, guiding the optimization according to their preferences. However, this brings up human fatigue problems caused by excessive interactions and solutions to evaluate. A common strategy to prevent this problem is limiting the number of interactions and solutions the DM evaluates. Machine Learning (ML) models were also used to learn how to evaluate solutions according to the DM profile and replace them after some interactions. Feature selection performs an essential task as non-relevant and/or redundant features used to train the ML model can reduce the accuracy and comprehensibility of the hypotheses induced by ML algorithms. This study aims to enhance the usage of an ML model in an interactive search-based PLA design approach by addressing two critical challenges: mitigating decision-maker fatigue through feature selection and improving computational efficiency, particularly during testing phases. We applied four selectors, and through results, We managed to reduce 30% of the features and 25% of the time spent on testing, achieving an accuracy of 99%.

Technical Debt of Software Projects Based on Merge Code Comments

Marcos Henrique Melo de Araújo, Catarina Costa, Awdren Fontão — Mon, 16 Jun 2025 00:00:00 +0000

Developers use code comments for various reasons, such as explaining code, documenting specifications, communicating with other developers, and highlighting upcoming tasks. Software projects with minimal documentation often have a significant number of comments. In this sense, the code comment analysis technique can be used to examine more complex aspects of software projects, such as TD generated by merge conflicts. The TD resulting from the resolution of merge conflicts occurs when the resulting code contains comments that indicate tasks to be performed in the future. There are no studies directly linking merge conflicts and TD. The purpose of this study is to identify and analyze code comments generated when resolving merge conflicts from this perspective. This process can lead to improvements in software quality and help in managing TD. To this end, an exploratory analysis was carried out in 100 software projects, with a specific focus on task annotations originating from the resolution of merge conflicts. The results revealed that 60.61% of the analyzed projects have at least one code comment indicating the creation or maintenance of TD. In addition, metrics such as accuracy, precision, recall, and F1-score were applied across different software projects to enhance the effectiveness and reliability of the built data model. The metric results suggest that the tool was able to correctly classify in most cases, but was not particularly precise in this classification due to the size of the analyzed projects and the variety of task annotations, including the presence of some and the absence of others.

Understanding How Feature Dependent Variables Affect Configurable System Comprehensibility

Djan Santos, Márcio Ribeiro, Cláudio Sant'Anna — Wed, 28 May 2025 00:00:00 +0000

Background: #ifdefs allow developers to define source code related to features that should or should not be compiled. A feature dependency occurs in a configurable system when source code snippets of different features share code elements, such as variables. Variables that produce feature dependency are called dependent variables. The dependency between two features may include just one dependent variable or more than one. It is reasonable to suspect that a high number of dependent variables and their use make the analysis of variability scenarios more complex. In fact, previous studies show that #ifdefs may affect comprehensibility, especially when their use implies feature dependency. Aims: In this sense, our goal is to understand how feature dependent variables affect the comprehensibility of configurable system source code. We conducted two complementary empirical studies. In Study 1, we evaluate if the comprehensibility of configurable system source code varies according to the number of dependent variables. Testing this hypothesis is important so that we can recommend practitioners and researchers the extent to which writing #ifdef code with dependencies is harmful. In study 2, we carried out an experiment in which developers analyzed programs with different degrees of variability. Our results show that the degree of variability did not affect the comprehensibility of programs with feature dependent variables. Method: We executed a controlled experiment with 12 participants who analyzed programs trying to specify their output. We quantified comprehensibility using metrics based on time and attempts to answer tasks correctly, participants’ visual effort, and participants’ heart rate. Results: Our results indicate that the higher the number of dependent variables the more difficult it was to understand programs with feature dependency. Conclusions: In practice, our results indicate that comprehensibility is more negatively affected in programs with higher number of dependent variables and when these variables are defined at a point far from the points where they are used.

Requirements Engineering for Machine Learning-Based AI Systems: A Tertiary Study

Fri, 05 Sep 2025 00:00:00 +0000

Context: In the last decade, machine learning (ML) components have become more and more present in contemporary software systems. A number of secondary literature studies reports challenges impacting on the development of ML-based systems, including those for requirements engineering (RE) activities. Motivation/Problem: Synthesizing secondary literature contributes to building knowledge and reaching conclusions about the existing RE approaches for ML-based systems (RE4ML), besides the novelty of a tertiary study on that subject. Objective: Through a tertiary study protocol we elaborated on, this paper synthesizes the body of evidence present in secondary studies on RE4ML systems. Method: We followed well-accepted guidelines about tertiary study protocols, including automatic search, the snowballing technique, selection and quality criteria, and data extraction and synthesis. Results: Nine secondary studies on RE4ML systems were aligned to our tertiary study's goal. We extracted and summarized the requirements elicitation, analysis, specification, validation, and management techniques for ML-based systems as well as the great challenges identified. Finally, we contribute with a nine-item research agenda to direct current and future searches to fill the gaps found. Conclusions: We conclude that RE has not been left aside in ML research, however, there are still challenges to be overcome, such as dealing with non-functional requirements, collaboration between stakeholders, and research in an industrial environment.

Comparative Analysis of Hoare Logic-Based Formal Verification Tools for Solidity Smart Contracts

Alexandre Mota, Manoel Villarim, Juliano Iyoda, Márcio Cornélio — Tue, 09 Dec 2025 00:00:00 +0000

Formal verification of smart contracts is widely regarded as an effective method for ensuring correctness and security properties across all possible executions. Its practical relevance has been driven by the availability of automatic verification tools that discharge intricate proofs. Another area of growing interest is the integration of specification paradigms - for example, combining Hoare-logic–style specifications (pre/postconditions and invariants) with SMT and symbolic reasoning - so that each technique can precisely capture complementary aspects of contract behavior. In this article we present a comparative analysis of four leading Solidity verification tools - solc-verify, SMTChecker, VeriSmart and the Certora Prover - and define what is meant here by a formal verification tool: a system that provides mathematically rigorous proofs that stated properties hold for every possible execution of a contract. We also describe a consistent evaluation framework that considers the Solidity version support, the preservation of the original contract structure, the local execution capability, the verification time, and the modeling-language requirements, among other criteria. We used the ERC-20 token standard as a benchmark and applied this framework to obtain empirical evidence of each tool’s capabilities and limitations. Our results expose substantial variability in the tools performances that undermines their trustworthiness in practice and highlights a gap between an academic tool capabilities and the industrial requirements. Finally, we discuss how these findings can inform developers and researchers in selecting appropriate verification tools, thereby contributing to improved smart contract security and reliability.

Natural Language Processing in Software Engineering: A Systematic Literature Review

Mon, 01 Dec 2025 00:00:00 +0000

Context: Software engineering (SE) artifacts and documents, such as requirements specifications, user stories, test cases, and concepts of operations (ConOps), are typically written in natural language, making their manipulation challenging. Natural Language Processing (NLP) is a viable solution for managing these tasks. Objective: To conduct a systematic literature review to explore the current use of NLP in SE artifacts and tasks, supplemented by a tertiary study focusing on the emerging role of Large Language Models (LLMs) in software engineering research. Method: We searched digital libraries for relevant papers and applied inclusion and exclusion criteria to filter the primary studies. We then analyzed NLP techniques applied to SE documents and examined their usage in this context. Our research methodology followed Kitchenham and Charters' guidelines. Additionally, we conducted a tertiary study to synthesize findings from existing systematic literature reviews and surveys specifically addressing LLMs in software engineering. Results: We selected 60 primary studies to identify the most common methods for NLP pipelines, feature extraction, language models, and machine learning algorithms used in SE. We also assessed the purposes of these methods, their benefits for SE, their difficulty, and their contribution to SE advancement. The tertiary study revealed a rapid proliferation of LLM-focused research, with comprehensive reviews documenting exponential growth in publications and widespread adoption across diverse SE tasks. Conclusion: Requirements are the most frequently addressed artifacts using NLP techniques, with preprocessing and part-of-speech (POS) tagging being widely used. There is a notable increase in the use of large language models for various SE tasks, such as requirements elicitation, source code generation, bug fixing, and software testing. The tertiary study confirms that LLMs represent a pivotal shift in the research landscape, warranting dedicated investigation to understand their transformative impact on NLP applications in software engineering.

From Detection to Refactoring of Microservice Bad Smells: A Systematic Literature Review

Mateus Dutra, Denis Pinheiro, Johnatan Oliveira, Eduardo Figueiredo — Mon, 27 Oct 2025 00:00:00 +0000

The extensive adoption of microservices architecture by technology companies is driven by its expected advantages, such as scalability, simplicity of development, and resilience, likely due to its cloud-native nature. However, the increasing complexity associated with this architecture can lead to the emergence of microservice smells, analogous to code smells, indicating potential architectural design issues. Despite the identification of numerous microservice smells in the literature, cohesive documentation to support architects detecting and refactoring them. This study conducts a Systematic Literature Review (SLR) to deepen the understanding of these microservice smells, explore detection tools and identify refactoring strategies to mitigate them. We conducted searches across six popular digital libraries, analyzing 27 relevant papers. As a result, we cataloged 104 distinct microservice bad smells, identified 7 detection tools, and compiled refactoring strategies for the most prevalent smells. This documentation aims to assist engineers and architects in identifying and effectively addressing microservice bad smells, thereby enhancing the quality and maintainability of microservice-based systems.

Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil: An Extended Analysis

Mon, 03 Nov 2025 00:00:00 +0000

[Context] In Brazil, 41% of companies use machine learning (ML) to some extent. However, several challenges have been reported when engineering ML-enabled systems, including unrealistic customer expectations and vagueness in ML problem specifications. Literature suggests that Requirements Engineering (RE) practices and tools may help to alleviate these issues, yet there is insufficient understanding of RE’s practical application and its perception among practitioners. [Goal] This study aims to investigate the application of RE in developing ML-enabled systems in Brazil, creating an overview of current practices, perceptions, and problems in the Brazilian industry. [Method] To this end, we extracted and analyzed data from an international survey focused on ML-enabled systems, concentrating specifically on responses from practitioners based in Brazil. We analyzed the cluster of RE-related answers gathered from 72 practitioners involved in data-driven projects. We conducted quantitative statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative studies on the reported problems involving open and axial coding procedures. [Results] Our findings highlight distinct RE implementation aspects in Brazil’s ML projects. For instance, (i) RE-related tasks are predominantly conducted by data scientists; (ii) the most common techniques for eliciting requirements are interviews and workshop meetings; (iii) there is a prevalence of interactive notebooks in requirements documentation; (iv) practitioners report problems that include a poor understanding of the problem to solve and the business domain, low customer engagement, and difficulties managing stakeholders expectations. Our analysis suggests that development methodology plays a role in these challenges. Agile methods appear to facilitate the management of customer expectations compared to traditional approaches; however, they also appear to introduce greater difficulties in problem understanding and customer involvement. [Conclusion] These results provide an understanding of RE-related practices and challenges in the Brazilian ML industry, helping to guide research and initiatives toward improving the maturity of RE for ML-enabled system projects.

Technical Debt in Pull Requests: Insights from Apache Projects

Felipe E. de O. Calixto, Eliane C. Araújo, Everton L. G. Alves — Mon, 03 Nov 2025 00:00:00 +0000

Technical Debt (TD) represents the effort required to address quality issues that affect a software system and progressively hinder code evolution over time. A pull request (PR) is a discrete unit of work that must meet specific quality standards to be integrated into the main codebase. PRs offer a valuable opportunity to assess how developers handle TD and how codebase quality evolves. In this work, we conducted two empirical analyses to understand how developers address TD within PRs and whether TD is effectively managed during PR reviews by both developers and reviewers. We examined 12 Java projects from Apache. The first study employed the SonarQube tool on 2,035 merged PRs to evaluate TD variation, identify the most frequently neglected and resolved types of TD issues, and analyze how TD evolves over time. The second study involved a qualitative analysis of review threads of 250 PRs, focusing on the types of PRs that frequently discuss TD, the characteristics of TD fix suggestions, and the reasons some suggestions are rejected. Our findings reveal that TD issues are prevalent in PRs, following a ratio of 1:2:1 (reduced: unchanged: increased). Among all TD issues, those related to code duplication and cognitive complexity are most frequently overlooked, while code duplication and obsolete code are the most commonly resolved. Regarding PR code review, we found that around 76% of review threads address TD, with code, design, and documentation being the most frequently discussed areas. Additionally, 96% of discussions include a fix suggestion, and over 80% of the discussed issues are resolved. These insights can help practitioners become more aware of TD management and may inspire the development of new tools to facilitate TD handling during PRs.

Impact and Insights of MBE Adoption in Knowledge Capture and Organization: A Survey of Brazilian Software Companies

André Menolli, Thiago Adriano Coleti, Edson OliveiraJr — Fri, 29 Aug 2025 00:00:00 +0000

Software development is a complex and knowledge-intensive process that involves multiple participants across various stages of the development lifecycle. Managing knowledge effectively in this context is challenging, especially when it comes to capturing, organizing, and reusing information in software engineering projects. Software development artifacts and techniques play a crucial role in addressing these challenges by facilitating the storage and sharing of knowledge. This study examines how model-based engineering facilitates various aspects of knowledge management. A survey of 62 Brazilian companies was conducted, providing a comprehensive roadmap for the future. Using qualitative and quantitative analyses, the findings were compared with those of other studies. The results indicate that different artifacts effectively support various knowledge management concepts across different phases of software development. Furthermore, while companies predominantly adopt artifact models in the early stages of development and recognize their benefits, they do not fully utilize their potential.

Transforming Interviews into Experiences: The Power of Immersive Role Playing in Requirements Engineering Education

Mon, 06 Oct 2025 00:00:00 +0000

Developing the necessary skills in Software Engineering students to conduct effective requirements elicitation interviews is a complex challenge. Immersive role playing has emerged as a promising educational strategy, enabling students to simulate realistic interviews, receive real-time feedback, and improve their performance under pressure. This approach blends traditional role playing with immersive learning environments, providing engaging and authentic experiences that better prepare students for industry demands. This article presents an experience report on the use of an immersive role playing to teach the interview technique for requirements elicitation. Conducted with 86 undergraduate students enrolled in a Requirements Engineering course, the study offers a broader perspective on the effectiveness and challenges of this approach. The findings suggest that immersive activities foster reflection, help identify areas for improvement, and emphasize the importance of emotional regulation in real-world interactions. These insights reinforce that mastering Requirements Engineering requires not only technical proficiency, but also strong interpersonal and emotional skills.

Software Tool Qualification: A Systematic Literature Review

Giovanni César Borges da Costa, Luiz Eduardo Galvão Martins — Sun, 23 Nov 2025 00:00:00 +0000

Safety-critical systems are subject to rigorous regulation, demanding compliance with specific criteria before it is made available. In safety-critical domains, this is accomplished through the fulfillment of recognized safety standards requirements or objectives. This SLR has the purpose of identifying approaches or methods to comply with safety standards requirements or objectives related to tool qualification aspects. It also pursued the identification of methods used to demonstrate tool confidence, as well as the activities and artifacts generated to demonstrate that. This study also identified challenges practitioners face to qualify software tools. This SLR was conducted per guidelines proposed by Kitchenham and Biolchini. A search string was developed, and its application returned more than 1,180 papers. For this study, 41 papers were selected, as per the criteria defined in the protocol. These SLR results indicate that there are many studies describing approaches based on established safety standards, for ensuring confidence in tool functionality. The results indicated that research on different proposed methods for tool qualification exists, especially in automotive context. The results also point to the necessity of additional research on toolchain integration. The SLR main findings were the identification of methods for tool qualification, concerns, approaches to address toolchain integration.

Remote Work in the Metaverse: How Software Development Teams Experience Gather.Town

Ayumi Aoki, Marcia Lima, Eriky Rodrigues, Leandro Galvão, Tayana Conte — Thu, 10 Jul 2025 00:00:00 +0000

The remote work model is becoming increasingly established in software companies, driven by the demand for flexibility and the advancement of communication technologies. Despite its benefits, this shift brings challenges, such as asynchronous communication, reduced opportunities for spontaneous interactions, and social isolation. To overcome these obstacles, some development teams are adopting metaverse environments, such as Gather.Town. In this context, this research investigates the impact of using metaverse environments on the experience of members of remote software development teams, as well as the factors that influence their adoption and abandonment. We conducted surveys and interviews with 13 practitioners from five software development companies who have used or are currently using Gather.Town. The study addressed aspects such as ease of use, communication, collaboration, sense of presence, social interaction, and flow experience, as well as adoption challenges and reasons for abandonment. Our results indicate that metaverse environments can facilitate collaboration in remote software development teams by promoting social interaction, enhancing the sense of presence, and enabling more efficient communication. This helps mitigate some of the limitations of traditional remote work, such as social isolation. However, participants also reported challenges, including distractions caused by playful elements and difficulties with privacy during sensitive interactions. Organizational restrictions were also identified as a barrier that limited the adoption of the platform. This research demonstrates that the use of metaverse environments can enhance communication and collaboration in remote teams. When properly aligned with the team's needs and workflows, these environments offer a more integrated and engaging work experience.

Analysis of the Role of GitHub Discussions as a Tool for Onboarding Newcomers in Open Source Software

Wed, 26 Nov 2025 00:00:00 +0000

The introduction of the discussions feature on GitHub provides a new, dedicated space for collaborative communication within open-source software (OSS) projects. This paper investigates the impact of discussions on community engagement, focusing on newcomer onboarding and the activity surrounding issues and pull requests. Through a comprehensive empirical analysis of 285 OSS projects, we observe a significant shift in participation patterns, with discussions emerging as a more attractive entry point for newcomers compared to traditional mechanisms like issues and pull requests. Our findings suggest that while discussions lower the barrier for newcomer participation, it leads to a decrease in traditional contributions such as new issues and pull requests. Additionally, we explore the engagement levels of different contributor roles, demonstrating how discussions foster diverse interactions and community involvement. The results provide critical insights into the evolving nature of collaboration on GitHub and highlight the role of discussions in shaping the dynamics of OSS project ecosystems.

Challenges and Solutions for Organizational Change in the Software Industry: A Tertiary Study

Rayfran Rocha Lima, Ana Carolina Oran, Leonardo Marques, César França — Mon, 27 Oct 2025 00:00:00 +0000

Organizational Change (OC) in the software industry is essential for maintaining competitiveness in dynamic environments. However, OC initiatives often face inefficiencies due to the lack of structured methodologies and reliance on ad hoc practices. This tertiary study analyzes 17 secondary studies related to the OC theme and its instances affecting key organizational pillars, such as technology, processes, and people. The results reveal limited adoption of OC models in software organizations. In response, it synthesizes 41 OC models, 14 critical success factors, and 6 key characteristics, offering a synthesized overview of structured OC frameworks and their relevance to the software industry context, supporting future adaptation efforts by practitioners.