-
Dynamic Test Case Prioritization in Industrial Test Result Datasets
Authors:
Alina Torbunova,
Per Erik Strandberg,
Ivan Porres
Abstract:
Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prio…
▽ More
Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prioritization schema that combines the use of a static and a dynamic prioritization algorithm. The dynamic prioritization algorithm rearranges the order of execution of tests on the fly, while the tests are being executed. We propose to use a conditional probability dynamic algorithm for this. We evaluate our solution on three industrial datasets and utilize Average Percentage of Fault Detection for that. The main findings are that our dynamic prioritization algorithm can: a) be applied with any static algorithm that assigns a priority score to each test case b) can improve the performance of the static algorithm if there are failure correlations between test cases c) can also reduce the performance of the static algorithm, but only when the static scheduling is performed at a near optimal level.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
The Westermo test system performance data set
Authors:
Per Erik Strandberg,
Yosh Marklund
Abstract:
There is a growing body of knowledge in the computer science, software engineering, software testing and software test automation disciplines. However, a challenge for researchers is to evaluate their research findings, ideas and tools due to lack of realistic data. This paper presents the Westermo test system performance data set. More than twenty performance metrics such as CPU and memory usage…
▽ More
There is a growing body of knowledge in the computer science, software engineering, software testing and software test automation disciplines. However, a challenge for researchers is to evaluate their research findings, ideas and tools due to lack of realistic data. This paper presents the Westermo test system performance data set. More than twenty performance metrics such as CPU and memory usage sampled twice per minute for a month on nineteen test systems driving nightly testing of cyber-physical systems has been anonymized and released. The industrial motivation is to spur work on anomaly detection in seasonal data such that one may increase trust in nightly testing. One could ask: If the test system is in an abnormal state - can we trust the test results? How could one automate the detection of abnormal states? The data set has previously been used by students and in hackathons. By releasing it we hope to simplify experiments on anomaly detection based on rules, thresholds, statistics, machine learning or artificial intelligence, perhaps while incorporating seasonality. We also hope that the data set could lead to findings in sustainable software engineering.
△ Less
Submitted 24 November, 2023;
originally announced November 2023.
-
Making Sense of Failure Logs in an Industrial DevOps Environment
Authors:
Muhammad Abbas,
Ali Hamayouni,
Mahshid Helali Moghadam,
Mehrdad Saadatmand,
Per Erik Strandberg
Abstract:
Processing and reviewing nightly test execution failure logs for large industrial systems is a tedious activity. Furthermore, multiple failures might share one root/common cause during test execution sessions, and the review might therefore require redundant efforts. This paper presents the LogGrouper approach for automated grou** of failure logs to aid root/common cause analysis and for enablin…
▽ More
Processing and reviewing nightly test execution failure logs for large industrial systems is a tedious activity. Furthermore, multiple failures might share one root/common cause during test execution sessions, and the review might therefore require redundant efforts. This paper presents the LogGrouper approach for automated grou** of failure logs to aid root/common cause analysis and for enabling the processing of each log group as a batch. LogGrouper uses state-of-art natural language processing and clustering approaches to achieve meaningful log grou**. The approach is evaluated in an industrial setting in both a qualitative and quantitative manner. Results show that LogGrouper produces good quality grou**s in terms of our two evaluation metrics (Silhouette Coefficient and Calinski-Harabasz Index) for clustering quality. The qualitative evaluation shows that experts perceive the groups as useful, and the groups are seen as an initial pointer for root cause analysis and failure assignment.
△ Less
Submitted 9 January, 2023;
originally announced January 2023.
-
The Westermo test results data set
Authors:
Per Erik Strandberg
Abstract:
There is a growing body of knowledge in the computer science, software engineering, software testing and software test automation disciplines. However, there is a challenge for researchers to evaluate their research findings, innovations and tools due to lack of realistic data. This paper presents the Westermo test results data set, more than one million verdicts from testing of embedded systems,…
▽ More
There is a growing body of knowledge in the computer science, software engineering, software testing and software test automation disciplines. However, there is a challenge for researchers to evaluate their research findings, innovations and tools due to lack of realistic data. This paper presents the Westermo test results data set, more than one million verdicts from testing of embedded systems, from more than five hundred consecutive days of nightly testing. The data also contains information on code changes in both the software under test and the test framework used for testing. This data set can support the research community in particular with respect to the regression test selection problem, flaky tests, test results visualization, etc.
△ Less
Submitted 24 November, 2022;
originally announced November 2022.
-
Industrial Requirements for Supporting AI-Enhanced Model-Driven Engineering
Authors:
Johan Bergelin,
Per Erik Strandberg
Abstract:
There is an increasing interest in research on the combination of AI techniques and methods with MDE. However, there is a gap between AI and MDE practices, as well as between researchers and practitioners. This paper tackles this gap by reporting on industrial requirements in this field. In the AIDOaRt research project, practitioners and researchers collaborate on AI-augmented automation supportin…
▽ More
There is an increasing interest in research on the combination of AI techniques and methods with MDE. However, there is a gap between AI and MDE practices, as well as between researchers and practitioners. This paper tackles this gap by reporting on industrial requirements in this field. In the AIDOaRt research project, practitioners and researchers collaborate on AI-augmented automation supporting modeling, coding, testing, monitoring, and continuous development in cyber-physical systems. The project specifically lies at the intersection of industry and academia collaboration with several industrial use cases. Through a process of elicitation and refinement, 78 high-level requirements were defined, and generalized into 30 generic requirements by the AIDOaRt partners. The main contribution of this paper is the set of generic requirements from the project for enhancing the development of cyber-physical systems with artificial intelligence, DevOps, and model-driven engineering, identifying the hot spots of industry needs in the interactions of MDE and AI. Future work will refine, implement and evaluate solutions toward these requirements in industry contexts.
△ Less
Submitted 29 August, 2022;
originally announced August 2022.
-
Automated System-Level Software Testing of Industrial Networked Embedded Systems
Authors:
Per Erik Strandberg
Abstract:
Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation.…
▽ More
Embedded systems are ubiquitous and play critical roles in management systems for industry and transport. Software failures in these domains may lead to loss of production or even loss of life, so the software in these systems needs to be reliable. Software testing is a standard approach for quality assurance of embedded software, and many software development processes strive for test automation. Out of the many challenges for successful software test automation, this thesis addresses five: (i) understanding how updated software reaches a test environment, how testing is conducted in the test environment, and how test results reach the developers that updated the software in the first place; (ii) selecting which test cases to execute in a test suite given constraints on available time and test systems; (iii) given that the test cases are run on different configurations of connected devices, selecting which hardware to use for each test case to be executed; (iv) analyzing test cases that, when executed over time on evolving software, testware or hardware revisions, appear to randomly fail; and (v) making test results information actionable with test results exploration and visualization. The challenges are tackled in several ways. [Abstract truncated.]
△ Less
Submitted 16 November, 2021;
originally announced November 2021.
-
Ethical AI-Powered Regression Test Selection
Authors:
Per Erik Strandberg,
Mirgita Frasheri,
Eduard Paul Enoiu
Abstract:
Test automation is common in software development; often one tests repeatedly to identify regressions. If the amount of test cases is large, one may select a subset and only use the most important test cases. The regression test selection (RTS) could be automated and enhanced with Artificial Intelligence (AI-RTS). This however could introduce ethical challenges. While such challenges in AI are in…
▽ More
Test automation is common in software development; often one tests repeatedly to identify regressions. If the amount of test cases is large, one may select a subset and only use the most important test cases. The regression test selection (RTS) could be automated and enhanced with Artificial Intelligence (AI-RTS). This however could introduce ethical challenges. While such challenges in AI are in general well studied, there is a gap with respect to ethical AI-RTS. By exploring the literature and learning from our experiences of develo** an industry AI-RTS tool, we contribute to the literature by identifying three challenges (assigning responsibility, bias in decision-making and lack of participation) and three approaches (explicability, supervision and diversity). Additionally, we provide a checklist for ethical AI-RTS to help guide the decision-making of the stakeholders involved in the process.
△ Less
Submitted 30 June, 2021;
originally announced June 2021.
-
The interacting nature of dwarf galaxies hosting superluminous supernovae
Authors:
Simon Vanggaard Ørum,
David Lykke Ivens,
Patrick Strandberg,
Giorgos Leloudas,
Allison W. S. Man,
Steve Schulze
Abstract:
(Abridged) Type I superluminous supernovae (SLSNe I) are rare, powerful explosions whose mechanism and progenitors remain elusive. SLSNe I show a preference for low-metallicity, actively star-forming dwarf galaxies. We investigate whether the hosts of SLSNe I show increased evidence for interaction. We use a sample of 42 SLSN I images obtained with $\textit{HST}$ and measure the number of companio…
▽ More
(Abridged) Type I superluminous supernovae (SLSNe I) are rare, powerful explosions whose mechanism and progenitors remain elusive. SLSNe I show a preference for low-metallicity, actively star-forming dwarf galaxies. We investigate whether the hosts of SLSNe I show increased evidence for interaction. We use a sample of 42 SLSN I images obtained with $\textit{HST}$ and measure the number of companion galaxies by counting the objects detected within a given radius from the host. As a comparison, we used two Monte Carlo-based methods to estimate the expected average number of companion objects in the same images, as well as a sample of 32 galaxies that have hosted long gamma-ray bursts (GRBs). About 50% of SLSN I hosts have at least one major companion (within a flux ratio of 1:4) within 5 kpc. The average number of major companions per SLSN I host galaxy is $0.70^{+0.19}_{-0.14}$. Our Monte Carlo comparison methods yield a lower number of companions for random objects of similar brightness in the same image or for the SLSN host after randomly redistributing the sources in the same image. The Anderson-Darling test shows that this difference is statistically significant independent of the redshift range. The same is true for the projected distance distribution of the companions. The SLSN I hosts are, thus, found in areas of their images, where the object number density is greater than average. SLSN I hosts have more companions than GRB hosts ($0.44^{+0.25}_{-0.13}$ companions per host distributed over 25% of the hosts) but the difference is not statistically significant. The difference between their separations is, however, marginally significant. The dwarf galaxies hosting SLSNe I are often part of interacting systems. This suggests that SLSNe I progenitors are formed after a recent burst of star formation. Low metallicity alone cannot explain this tendency.
△ Less
Submitted 15 September, 2020; v1 submitted 8 September, 2020;
originally announced September 2020.
-
Intermittently Failing Tests in the Embedded Systems Domain
Authors:
Per Erik Strandberg,
Thomas J Ostrand,
Elaine J Weyuker,
Wasif Afzal,
Daniel Sundmark
Abstract:
Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of th…
▽ More
Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.
△ Less
Submitted 14 May, 2020;
originally announced May 2020.
-
Ethical Interviews in Software Engineering
Authors:
Per Erik Strandberg
Abstract:
Background: Despite a long history, numerous laws and regulations, ethics remains an unnatural topic for many software engineering researchers. Poor research ethics may lead to mistrust of research results, lost funding and retraction of publications. A core principle for research ethics is confidentiality, and anonymization is a standard approach to guarantee it. Many guidelines for qualitative s…
▽ More
Background: Despite a long history, numerous laws and regulations, ethics remains an unnatural topic for many software engineering researchers. Poor research ethics may lead to mistrust of research results, lost funding and retraction of publications. A core principle for research ethics is confidentiality, and anonymization is a standard approach to guarantee it. Many guidelines for qualitative software engineering research, and for qualitative research in general, exist, but these do not penetrate how and why to anonymize interview data. Aims: In this paper we aim to identify ethical guidelines for software engineering interview studies involving industrial practitioners. Method: By learning from previous experiences and listening to the authority of existing guidelines in the more mature field of medicine as well as in software engineering, a comprehensive set of checklists for interview studies was distilled. Results: The elements of an interview study were identified and ethical considerations and recommendations for each step were produced, in particular with respect to anonymization. Important ethical principles are: consent, beneficence, confidentiality, scientific value, researcher skill, justice, respect for law, and ethical reviews. Conclusions: The most important contribution of this study is the set of checklists for ethical interview studies. Future work is needed to refine these guidelines with respect to legal aspects and ethical boards.
△ Less
Submitted 19 June, 2019;
originally announced June 2019.
