Background. The popularity of tools for analyzing Technical Debt, and particularly that of SonarQube, is increasing rapidly. SonarQube proposes a set of coding rules, which represent something wrong in the code that will soon be reflected in a fault or will increase maintenance effort. However, while the management of some companies is encouraging developers not to violate these rules in the first place and to produce code below a certain technical debt threshold, developers are skeptical of their importance. Objective. In order to understand which SonarQube violations are actually fault-prone and to analyze the accuracy of the fault-prediction model, we designed and conducted an empirical study on 21 well-known mature open-source projects. Method. We applied the SZZ algorithm to label the fault-inducing commits. We compared the classification power of eight Machine Learning models (Logistic Regression, Decision Tree, Random Forest, Extremely Randomized Trees, AdaBoost, Gradient Boosting, XGBoost) to obtain a set of violations that are correlated with fault-inducing commits. Finally, we calculated the percentage of violations introduced in the fault-inducing commit and removed in the fault-fixing commit, so as to reduce the risk of spurious correlations. Result. Among the 202 violations defined for Java by SonarQube, only 26 have a relatively low fault-proneness. Moreover, violations classified as "bugs" by SonarQube hardly never become a failure. Consequently, the accuracy of the fault-prediction power proposed by SonarQube is extremely low. Conclusion. The rules applied by SonarQube for calculating technical debt should be thoroughly investigated and their harmfulness needs to be further confirmed. Therefore, companies should carefully consider which rules they really need to apply, especially if their goal is to reduce faultproneness. The popularity of tools for analyzing technical debt, such as SonarQube, is increasing rapidly. In particular, SonarQube has been adopted by more than 85K organizations 1 including nearly 15K public open-source projects 2 . SonarQube analyzes code compliance against a set of rules. If the code violates some rule, SonarQube adds the time needed to refactor the violated rule as part of the technical debt. SonarQube also identifies a set of rules as "bugs", claiming that they "represent something wrong in the code and will soon be reflected in a fault"; moreover, they also claim that zero false positives are expected from "bugs" 3 . Although SonarQube recommends customizing the out-of-the-box set of rules (named "sonar way") 4 , practitioners are reluctant to customize it and commonly rely on the "sonar way" standard rule-set [43] . Querying the SonarQube public instance APIs 5 , we can see that more than 98% of the public projects (14,732 projects up to 14,957) uses the "sonar way" rule set. However, even if developers are not sure about the usefulness of the rules [43], [12] , they do pay attention to different rules categories and priorities and remove violations related to rules with high severity [43] in order to avoid the possible risk of faults [12] . Therefore, investing effort for tasks that do not bring the expected benefits. Several studies have analyzed the impact of code smells [16] on faults [41], [45], [25][34]. At best, only a limited number of studies have considered SonarQube rule violations [14], [26], but they did not investigate the impact of the SonarQube violations considered as "bugs" on faults. The goal of this work is twofold: -Analyze the fault-proneness of SonarQube rule violations, and in particular, understand if rules classified as "bugs" are more fault-prone than security and maintainability rules. -Analyze the accuracy of the quality model provided by SonarQube in order to understand the fault-prediction accuracy of the rules classified as "bugs".
Lenarduzzi et al. REF investigated the fault proneness of SonarQube violations, in order to understand which violations are actually fault-prone and to assess the fault-prediction model accuracy.
On the Fault Proneness of SonarQube Technical Debt Violations: A comparison of eight Machine Learning Techniques.
ArXiv
ArXiv
Computer Science