-
AI-Assisted Assessment of Coding Practices in Modern Code Review
Authors:
Manushree Vijayvergiya,
Małgorzata Salawa,
Ivan Budiselić,
Dan Zheng,
Pascal Lamblin,
Marko Ivanković,
Juanjo Carin,
Mateusz Lewko,
Jovan Andonov,
Goran Petrović,
Daniel Tarlow,
Petros Maniatis,
René Just
Abstract:
Modern code review is a process in which an incremental code contribution made by a code author is reviewed by one or more peers before it is committed to the version control system. An important element of modern code review is verifying that code contributions adhere to best practices. While some of these best practices can be automatically verified, verifying others is commonly left to human re…
▽ More
Modern code review is a process in which an incremental code contribution made by a code author is reviewed by one or more peers before it is committed to the version control system. An important element of modern code review is verifying that code contributions adhere to best practices. While some of these best practices can be automatically verified, verifying others is commonly left to human reviewers. This paper reports on the development, deployment, and evaluation of AutoCommenter, a system backed by a large language model that automatically learns and enforces coding best practices. We implemented AutoCommenter for four programming languages (C++, Java, Python, and Go) and evaluated its performance and adoption in a large industrial setting. Our evaluation shows that an end-to-end system for learning and enforcing coding best practices is feasible and has a positive impact on the developer workflow. Additionally, this paper reports on the challenges associated with deploying such a system to tens of thousands of developers and the corresponding lessons learned.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
MuRS: Mutant Ranking and Suppression using Identifier Templates
Authors:
Zimin Chen,
Malgorzata Salawa,
Manushree Vijayvergiya,
Goran Petrovic,
Marko Ivankovic,
Rene Just
Abstract:
Diff-based mutation testing is a mutation testing approach that only mutates lines affected by a code change under review. Google's mutation testing service integrates diff-based mutation testing into the code review process and continuously gathers developer feedback on mutants surfaced during code review. To enhance the developer experience, the mutation testing service implements a number of su…
▽ More
Diff-based mutation testing is a mutation testing approach that only mutates lines affected by a code change under review. Google's mutation testing service integrates diff-based mutation testing into the code review process and continuously gathers developer feedback on mutants surfaced during code review. To enhance the developer experience, the mutation testing service implements a number of suppression rules, which target not-useful mutants-that is, mutants that have consistently received negative developer feedback. However, while effective, manually implementing suppression rules require significant engineering time. An automatic system to rank and suppress mutants would facilitate the maintenance of the mutation testing service. This paper proposes and evaluates MuRS, an automated approach that groups mutants by patterns in the source code under test and uses these patterns to rank and suppress future mutants based on historical developer feedback on mutants in the same group. To evaluate MuRS, we conducted an A/B testing study, comparing MuRS to the existing mutation testing service. Despite the strong baseline, which uses manually developed suppression rules, the results show a statistically significantly lower negative feedback ratio of 11.45% for MuRS versus 12.41% for the baseline. The results also show that MuRS is able to recover existing suppression rules implemented in the baseline. Finally, the results show that statement-deletion mutant groups received both the most positive and negative developer feedback, suggesting a need for additional context that can distinguish between useful and not-useful mutants in these groups. Overall, MuRS has the potential to substantially reduce the development and maintenance cost for an effective mutation testing service by automatically learning suppression rules.
△ Less
Submitted 15 June, 2023;
originally announced June 2023.
-
Analytic Solutions for Wheeled Mobile Manipulator Supporting Forces
Authors:
Goran R. Petrović,
Jouni Mattila
Abstract:
When a mobile manipulator's wheel loses contact with the ground, tip**-over may occur, causing material damage, and in the worst case, it can put human lives in danger. The tip-over stability of wheeled mobile manipulators must not be overlooked at any stage of a mobile manipulator's life, starting from the design phase, continuing through the commissioning period and extending to the operationa…
▽ More
When a mobile manipulator's wheel loses contact with the ground, tip**-over may occur, causing material damage, and in the worst case, it can put human lives in danger. The tip-over stability of wheeled mobile manipulators must not be overlooked at any stage of a mobile manipulator's life, starting from the design phase, continuing through the commissioning period and extending to the operational phase. Many tip-over stability criteria formulated throughout the years do not explicitly consider the normal wheel loads, with most of them relying on prescribed stability margins in terms of overturning moments. In these formulations, it is commonly argued that overturning will occur about one of the axes connecting adjacent manipulator's contact points with the ground. This claim may not always be valid and is certainly restrictive. Explicit expressions for the manipulator supporting forces provide the best insight into relevant affecting terms which contribute to the tip-over (in)stability. They also remove the necessity for thinking about which axis the manipulator could tip over and simultaneously enable the formulation of more intuitive stability margins and on-line tip-over prevention techniques. The present study presents a general dynamics modelling approach in the Newton--Euler framework using 6D vectors and gives normal wheel load equations in a typical 4-wheeled mobile manipulator negotiating a slope. The given expressions are expected to become standard in wheeled mobile manipulators and to provide a basis for effective tip-over stability criteria and tip-over avoidance techniques. Based on the presented results, specific improvements of the state-of-the-art criteria are discussed.
△ Less
Submitted 21 April, 2022; v1 submitted 21 February, 2022;
originally announced February 2022.
-
Mathematical modelling and virtual decomposition control of heavy-duty parallel$-$serial hydraulic manipulators
Authors:
Goran R. Petrović,
Jouni Mattila
Abstract:
This paper proposes a novel modelling approach for a heavy-duty manipulator with parallel$-$serial structures connected in series. Each considered parallel$-$serial structure contains a revolute segment with rigid links connected by a passive revolute joint and actuated by a linear hydraulic actuator, thus forming a closed kinematic loop. In addition, prismatic segments, consisting of prismatic jo…
▽ More
This paper proposes a novel modelling approach for a heavy-duty manipulator with parallel$-$serial structures connected in series. Each considered parallel$-$serial structure contains a revolute segment with rigid links connected by a passive revolute joint and actuated by a linear hydraulic actuator, thus forming a closed kinematic loop. In addition, prismatic segments, consisting of prismatic joints driven by hydraulic linear actuators, also are considered. Expressions for actuator forces are derived using the Newton$-$Euler (N$-$E) dynamics formulation. The derivation process does not assume massless actuators decoupled from manipulator links, which is common in the Lagrange dynamics formulation. Actuator pressure dynamics are included in the analysis, leading in total to a third-order system of ordinary differential equations (ODEs). The proposed model in the N$-$E framework, with fewer parameters than its predecessors, inspires revision of the virtual decomposition control (VDC) systematic process to formulate a control law based on the new model. The virtual stability of each generic manipulator revolute and prismatic segment is obtained, leading to the Lyapunov stability of the entire robot.
△ Less
Submitted 10 January, 2022; v1 submitted 11 August, 2021;
originally announced August 2021.
-
Does mutation testing improve testing practices?
Authors:
Goran Petrović,
Marko Ivanković,
Gordon Fraser,
René Just
Abstract:
Various proxy metrics for test quality have been defined in order to guide developers when writing tests. Code coverage is particularly well established in practice, even though the question of how coverage relates to test quality is a matter of ongoing debate. Mutation testing offers a promising alternative: Artificial defects can identify holes in a test suite, and thus provide concrete suggesti…
▽ More
Various proxy metrics for test quality have been defined in order to guide developers when writing tests. Code coverage is particularly well established in practice, even though the question of how coverage relates to test quality is a matter of ongoing debate. Mutation testing offers a promising alternative: Artificial defects can identify holes in a test suite, and thus provide concrete suggestions for additional tests. Despite the obvious advantages of mutation testing, it is not yet well established in practice. Until recently, mutation testing tools and techniques simply did not scale to complex systems. Although they now do scale, a remaining obstacle is lack of evidence that writing tests for mutants actually improves test quality. In this paper we aim to fill this gap: By analyzing a large dataset of almost 15 million mutants, we investigate how these mutants influenced developers over time, and how these mutants relate to real faults. Our analyses suggest that developers using mutation testing write more tests, and actively improve their test suites with high quality tests such that fewer mutants remain. By analyzing a dataset of past fixes of real high-priority faults, our analyses further provide evidence that mutants are indeed coupled with real faults. In other words, had mutation testing been used for the changes introducing the faults, it would have reported a live mutant that could have prevented the bug.
△ Less
Submitted 12 March, 2021;
originally announced March 2021.
-
Practical Mutation Testing at Scale
Authors:
Goran Petrović,
Marko Ivanković,
Gordon Fraser,
René Just
Abstract:
Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing ha…
▽ More
Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem -- both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 500,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas: (1) Mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base; (2) Mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) Mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. Evaluation in a code-review-based setting with more than 24,000 developers on more than 1,000 projects shows that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable up to large-scale industrial settings.
△ Less
Submitted 26 February, 2021; v1 submitted 22 February, 2021;
originally announced February 2021.
