Showing 1–8 of 8 results for author: Cornejo, O

Fuzzing for CPS Mutation Testing

Authors: Jaekwon Lee, Enrico Viganò, Oscar Cornejo, Fabrizio Pastore, Lionel Briand

Abstract: Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data generation techniques for mutation testing of C and C++ software, two typical languages for CPS software, rely on symbolic execution, whose limit… ▽ More Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data generation techniques for mutation testing of C and C++ software, two typical languages for CPS software, rely on symbolic execution, whose limitations often prevent its application (e.g., it cannot test black-box components). We propose a mutation testing approach that leverages fuzz testing, which has proved effective with C and C++ software. Fuzz testing automatically generates diverse test inputs that exercise program branches in a varied number of ways and, therefore, exercise statements in different program states, thus maximizing the likelihood of killing mutants, our objective. We performed an empirical assessment of our approach with software components used in satellite systems currently in orbit. Our empirical evaluation shows that mutation testing based on fuzz testing kills a significantly higher proportion of live mutants than symbolic execution (i.e., up to an additional 47 percentage points). Further, when symbolic execution cannot be applied, fuzz testing provides significant benefits (i.e., up to 41% mutants killed). Our study is the first one comparing fuzz testing and symbolic execution for mutation testing; our results provide guidance towards the development of fuzz testing tools dedicated to mutation testing. △ Less

Submitted 15 August, 2023; originally announced August 2023.

Comments: This article is the camera-ready version for ASE 2023

Data-driven Mutation Analysis for Cyber-Physical Systems

Authors: Enrico Viganò, Oscar Cornejo, Fabrizio Pastore, Lionel Briand

Abstract: Cyber-physical systems (CPSs) typically consist of a wide set of integrated, heterogeneous components; consequently, most of their critical failures relate to the interoperability of such components.Unfortunately, most CPS test automation techniques are preliminary and industry still heavily relies on manual testing. With potentially incomplete, manually-generated test suites, it is of paramount i… ▽ More Cyber-physical systems (CPSs) typically consist of a wide set of integrated, heterogeneous components; consequently, most of their critical failures relate to the interoperability of such components.Unfortunately, most CPS test automation techniques are preliminary and industry still heavily relies on manual testing. With potentially incomplete, manually-generated test suites, it is of paramount importance to assess their quality. Though mutation analysis has demonstrated to be an effective means to assess test suite quality in some specific contexts, we lack approaches for CPSs. Indeed, existing approaches do not target interoperability problems and cannot be executed in the presence of black-box or simulated components, a typical situation with CPSs. In this paper, we introduce data-driven mutation analysis, an approach that consists in assessing test suite quality by verifying if it detects interoperability faults simulated by mutating the data exchanged by software components. To this end, we describe a data-driven mutation analysis technique (DaMAT) that automatically alters the data exchanged through data buffers. Our technique is driven by fault models in tabular form where engineers specify how to mutate data items by selecting and configuring a set of mutation operators. We have evaluated DaMAT with CPSs in the space domain; specifically, the test suites for the software systems of a microsatellite and nanosatellites launched on orbit last year. Our results show that the approach effectively detects test suite shortcomings, is not affected by equivalent and redundant mutants, and entails acceptable costs. △ Less

Submitted 5 October, 2022; v1 submitted 25 January, 2022; originally announced January 2022.

Comments: 19 pages

Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

Authors: Oscar Cornejo, Fabrizio Pastore, Lionel Briand

Abstract: On-board embedded software developed for spaceflight systems (space software) must adhere to stringent software quality assurance procedures. For example, verification and validation activities are typically performed and assessed by third party organizations. To further minimize the risk of human mistakes, space agencies, such as the European Space Agency (ESA), are looking for automated solution… ▽ More On-board embedded software developed for spaceflight systems (space software) must adhere to stringent software quality assurance procedures. For example, verification and validation activities are typically performed and assessed by third party organizations. To further minimize the risk of human mistakes, space agencies, such as the European Space Agency (ESA), are looking for automated solutions for the assessment of software testing activities, which play a crucial role in this context. Over the years, mutation analysis has shown to be a promising solution for the automated assessment of test suites; it consists of measuring the quality of a test suite in terms of the percentage of injected faults leading to a test failure. A number of optimization techniques, addressing scalability and accuracy problems, have been proposed to facilitate the industrial adoption of mutation analysis. However, to date, two major problems prevent space agencies from enforcing mutation analysis in space software development. In this paper, we enhance mutation analysis optimization techniques to enable their applicability to embedded software and propose a pipeline that successfully integrates them to address scalability and accuracy issues in this context, as described above. Further, we report on the largest study involving embedded software systems in the mutation analysis literature. Our research is part of a research project funded by ESA ESTEC involving private companies (GomSpace Luxembourg and LuxSpace) in the space sector. These industry partners provided the case studies reported in this paper; they include an on-board software system managing a microsatellite currently on-orbit, a set of libraries used in deployed cubesats, and a mathematical library certified by ESA. △ Less

Submitted 23 August, 2021; v1 submitted 13 January, 2021; originally announced January 2021.

Comments: Accepted for publication on IEEE TRANSACTIONS ON SOFTWARE ENGINEERING

CBR: Controlled Burst Recording

Authors: Oscar Cornejo, Daniela Briola, Daniela Micucci, Leonardo Mariani

Abstract: Collecting traces from software running in the field is both useful and challenging. Traces may indeed help revealing unexpected usage scenarios, detecting and reproducing failures, and building behavioral models that reflect how the software is actually used. On the other hand, recording traces is an intrusive activity that may annoy users, negatively affecting the usability of the applications,… ▽ More Collecting traces from software running in the field is both useful and challenging. Traces may indeed help revealing unexpected usage scenarios, detecting and reproducing failures, and building behavioral models that reflect how the software is actually used. On the other hand, recording traces is an intrusive activity that may annoy users, negatively affecting the usability of the applications, if not properly designed. In this paper we address field monitoring by introducing Controlled Burst Recording, a monitoring solution that can collect comprehensive runtime data without compromising the quality of the user experience. The technique encodes the knowledge extracted from the monitored application as a finite state model that both represents the sequences of operations that can be executed by the users and the corresponding internal computations that might be activated by each operation. Our initial assessment with information extracted from ArgoUML shows that Controlled Burst Recording can reconstruct behavioral information more effectively than competing sampling techniques, with a low impact on the system response time. △ Less

Submitted 8 February, 2020; v1 submitted 5 February, 2020; originally announced February 2020.

Comments: accepted at ICST2020 https://icst2020.info/

MSC Class: 68 ACM Class: D.2

In-The-Field Monitoring of Functional Calls: Is It Feasible?

Authors: Oscar Cornejo, Daniela Briola, Daniela Micucci, Leonardo Mariani

Abstract: Collecting data about the sequences of function calls executed by an application while running in the field can be useful to a number of applications, including failure reproduction, profiling, and debugging. Unfortunately, collecting data from the field may introduce annoying slowdowns that negatively affect the quality of the user experience. So far, the impact of monitoring has been mainly stud… ▽ More Collecting data about the sequences of function calls executed by an application while running in the field can be useful to a number of applications, including failure reproduction, profiling, and debugging. Unfortunately, collecting data from the field may introduce annoying slowdowns that negatively affect the quality of the user experience. So far, the impact of monitoring has been mainly studied in terms of the overhead that it may introduce in the monitored applications, rather than considering if the introduced overhead can be really recognized by users. In this paper we take a different perspective studying to what extent collecting data about sequences of function calls may impact the quality of the user experience, producing recognizable effects. Interestingly we found that, depending on the nature of the executed operation and its execution context, users may tolerate a non-trivial overhead. This information can be potentially exploited to collect significant amount of data without annoying users. △ Less

Submitted 20 January, 2020; originally announced January 2020.

MSC Class: 68 ACM Class: D.2

Fragmented Monitoring

Authors: Oscar Cornejo, Daniela Briola, Daniela Micucci, Leonardo Mariani

Abstract: Field data is an invaluable source of information for testers and developers because it witnesses how software systems operate in real environments, capturing scenarios and configurations relevant to end-users. Unfortunately, collecting traces might be resource-consuming and can significantly affect the user experience, for instance causing annoying slowdowns. Existing monitoring techniques ca… ▽ More Field data is an invaluable source of information for testers and developers because it witnesses how software systems operate in real environments, capturing scenarios and configurations relevant to end-users. Unfortunately, collecting traces might be resource-consuming and can significantly affect the user experience, for instance causing annoying slowdowns. Existing monitoring techniques can control the overhead introduced in the applications by reducing the amount of collected data, for instance by collecting each event only with a given probability. However, collecting fewer events limits the amount of information extracted from the field and may fail in providing a comprehensive picture of the behavior of a program. In this paper we present fragmented monitoring, a monitoring technique that addresses the issue of collecting information from the field without annoying users. The key idea of fragmented monitoring is to reduce the overhead by recording partial traces (fragments) instead of full traces, while annotating the beginning and the end of each fragment with state information. These annotations are exploited offline to derive traces that might be likely observed in the field and that could not be collected directly due to the overhead that would be introduced in a program. △ Less

Submitted 23 August, 2017; originally announced August 2017.

Comments: In Proceedings PrePost 2017, arXiv:1708.06889

ACM Class: D.2.5

Journal ref: EPTCS 254, 2017, pp. 57-68

Flexible In-The-Field Monitoring

Authors: Oscar Cornejo

Abstract: Fully assessing the robustness of a software application in-house is infeasible, especially considering the huge variety of hardly predictable stimuli, environments, and configurations that applications must handle in the field. For this reason, modern testing and analysis techniques can often process data extracted from the field, such as crash reports and profile data, or can even be executed di… ▽ More Fully assessing the robustness of a software application in-house is infeasible, especially considering the huge variety of hardly predictable stimuli, environments, and configurations that applications must handle in the field. For this reason, modern testing and analysis techniques can often process data extracted from the field, such as crash reports and profile data, or can even be executed directly in the field, for instance to diagnose and correct problems. In all these cases, collection, processing, and distribution of field data must be done seamlessly and unobstrusively while users interact with their applications. To limit the intrusiveness of in-the-field monitoring a common approach is to reduce the amount of collected data (e.g., to rare events and to crash dumps), which, however, may severely affect the effectiveness of the techniques that exploit field data. The objective of this Ph.D. thesis is to define solutions for collecting field data in a cost effective way without affecting the quality of the user experience. This result can enable a new range of testing and analysis solutions that extensively exploit field data. △ Less

Submitted 20 May, 2017; originally announced May 2017.

Comments: 2 pages, no figures, conference article

In The Field Monitoring of Interactive Applications

Authors: Oscar Cornejo, Daniela Briola, Daniela Micucci, Leonardo Mariani

Abstract: Monitoring techniques can extract accurate data about the behavior of software systems. When used in the field, they can reveal how applications behave in real-world contexts and how programs are actually exercised by their users. Nevertheless, since monitoring might need significant storage and computational resources, it may interfere with users activities degrading the quality of the user exper… ▽ More Monitoring techniques can extract accurate data about the behavior of software systems. When used in the field, they can reveal how applications behave in real-world contexts and how programs are actually exercised by their users. Nevertheless, since monitoring might need significant storage and computational resources, it may interfere with users activities degrading the quality of the user experience. While the impact of monitoring has been typically studied by measuring the overhead that it may introduce in a monitored application, there is little knowledge about how monitoring solutions may actually impact on the user experience and to what extent users may recognize their presence. In this paper, we present our investigation on how collecting data in the field may impact the quality of the user experience. Our initial results show that non-trivial overhead can be tolerated by users, depending on the kind of activity that is performed. This opens interesting opportunities for research in monitoring solutions, which could be designed to opportunistically △ Less

Submitted 18 May, 2017; originally announced May 2017.

Comments: IEEE Copyright, Accepted for pubblication in ICSE 2017, Nier Track