Showing 1–6 of 6 results for author: Čaušević, A

Exploring Behaviours of RESTful APIs in an Industrial Setting

Authors: Stefan Karlsson, Robbert Jongeling, Adnan Causevic, Daniel Sundmark

Abstract: A common way of exposing functionality in contemporary systems is by providing a Web-API based on the REST API architectural guidelines. To describe REST APIs, the industry standard is currently OpenAPI-specifications. Test generation and fuzzing methods targeting OpenAPI-described REST APIs have been a very active research area in recent years. An open research challenge is to aid users in better… ▽ More A common way of exposing functionality in contemporary systems is by providing a Web-API based on the REST API architectural guidelines. To describe REST APIs, the industry standard is currently OpenAPI-specifications. Test generation and fuzzing methods targeting OpenAPI-described REST APIs have been a very active research area in recent years. An open research challenge is to aid users in better understanding their API, in addition to finding faults and to cover all the code. In this paper, we address this challenge by proposing a set of behavioural properties, common to REST APIs, which are used to generate examples of behaviours that these APIs exhibit. These examples can be used both (i) to further the understanding of the API and (ii) as a source of automatic test cases. Our evaluation shows that our approach can generate examples deemed relevant for understanding the system and for a source of test generation by practitioners. In addition, we show that basing test generation on behavioural properties provides tests that are less dependent on the state of the system, while at the same time yielding a similar code coverage as state-of-the-art methods in REST API fuzzing in a given time limit. △ Less

Submitted 26 October, 2023; originally announced October 2023.

Exploring API Behaviours Through Generated Examples

Authors: Stefan Karlsson, John Hughes, Robbert Jongeling, Adnan Causevic, Daniel Sundmark

Abstract: Understanding the behaviour of a system's API can be hard. Giving users access to relevant examples of how an API behaves has been shown to make this easier for them. In addition, such examples can be used to verify expected behaviour or identify unwanted behaviours. Methods for automatically generating examples have existed for a long time. However, state-of-the-art methods rely on either white… ▽ More Understanding the behaviour of a system's API can be hard. Giving users access to relevant examples of how an API behaves has been shown to make this easier for them. In addition, such examples can be used to verify expected behaviour or identify unwanted behaviours. Methods for automatically generating examples have existed for a long time. However, state-of-the-art methods rely on either white-box information, such as source code, or on formal specifications of the system behaviour. But what if you do not have access to either? e.g., when interacting with a third-party API. In this paper, we present an approach to automatically generate relevant examples of behaviours of an API, without requiring either source code or a formal specification of behaviour. Evaluation on an industry-grade REST API shows that our method can produce small and relevant examples that can help engineers to understand the system under exploration. △ Less

Submitted 29 August, 2023; originally announced August 2023.

Automatic Property-based Testing of GraphQL APIs

Authors: Stefan Karlsson, Adnan Čaušević, Daniel Sundmark

Abstract: In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs. In this paper, we propose a method for automatically prod… ▽ More In recent years, GraphQL has become a popular way to expose web APIs. With its raise of adoption in industry, the quality of GraphQL APIs must be also assessed, as with any part of a software system, and preferably in an automated manner. However, there is currently a lack of methods to automatically generate tests to exercise GraphQL APIs. In this paper, we propose a method for automatically producing GraphQL queries to test GraphQL APIs. This is achieved using a property-based approach to create a generator for queries based on the GraphQL schema of the system under test. Our evaluation on a real world software system shows that this approach is both effective, in terms of finding real bugs, and efficient, as a complete schema can be covered in seconds. In addition, we evaluate the fault finding capability of the method when seeding known faults. 73% of the seeded faults where found, with room for improvements with regards to domain specific behavior, a common oracle challenge in automatic test generation. △ Less

Submitted 14 December, 2020; originally announced December 2020.

Model-based Automated Testing of Mobile Applications: An Industrial Case Study

Authors: Stefan Karlsson, Adnan Čaušević, Daniel Sundmark, Mårten Larsson

Abstract: Automatic testing of mobile applications has been a well-researched area in recent years. However, testing in industry is still a very manual practice, as research results have not been fully transferred and adopted. Considering mobile applications, manual testing has the additional burden of adequate testing posed by a large number of available devices and different configurations, as well as the… ▽ More Automatic testing of mobile applications has been a well-researched area in recent years. However, testing in industry is still a very manual practice, as research results have not been fully transferred and adopted. Considering mobile applications, manual testing has the additional burden of adequate testing posed by a large number of available devices and different configurations, as well as the maintenance and setup of such devices. In this paper, we propose and evaluate the use of a model-based test generation approach, where generated tests are executed on a set of cloud-hosted real mobile devices. By using a model-based approach we generate dynamic, less brittle, and implementation simple test cases. The test execution on multiple real devices with different configurations increase the confidence in the implementation of the system under test. Our evaluation shows that the used approach produces a high coverage of the parts of the application related to user interactions. Nevertheless, the inclusion of external services in test generation is required in order to additionally increase the coverage of the complete application. Furthermore, we present the lessons learned while transferring and implementing this approach in an industrial context and applying it to the real product. △ Less

Submitted 20 August, 2020; originally announced August 2020.

QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs

Authors: Stefan Karlsson, Adnan Causevic, Daniel Sundmark

Abstract: RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods needs to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic p… ▽ More RESTful APIs are an increasingly common way to expose software systems functionality and it is therefore of high interest to find methods to automatically test and verify such APIs. To lower the barrier for industry adoption, such methods needs to be straightforward to use with a low effort. This paper introduces a method to explore the behaviour of a RESTful API. This is done by using automatic property-based tests produced from OpenAPI documents that describe the REST API under test. We describe how this method creates artifacts that can be leveraged both as property-based test generators and as a source of validation for results (i.e., as test oracles). Experimental results, on both industrial and open source services, indicate how this approach is a low effort way of finding real faults. Furthermore, it supports building additional knowledge about the system under test by automatically exposing misalignment of specification and implementation. Since the tests are generated from the OpenAPI document this method automatically evolves test cases as the REST API evolves. △ Less

Submitted 20 December, 2019; originally announced December 2019.

Enablers and Impediments for Collaborative Research in Software Testing: An Empirical Exploration

Authors: Eduard Paul Enoiu, Adnan Causevic

Abstract: When it comes to industrial organizations, current collaboration efforts in software engineering research are very often kept in-house, depriving these organizations off the skills necessary to build independent collaborative research. The current trend, towards empirical software engineering research, requires certain standards to be established which would guide these collaborative efforts in cr… ▽ More When it comes to industrial organizations, current collaboration efforts in software engineering research are very often kept in-house, depriving these organizations off the skills necessary to build independent collaborative research. The current trend, towards empirical software engineering research, requires certain standards to be established which would guide these collaborative efforts in creating a strong partnership that promotes independent, evidence-based, software engineering research. This paper examines key enabling factors for an efficient and effective industry-academia collaboration in the software testing domain. A major finding of the research was that while technology is a strong enabler to better collaboration, it must be complemented with industrial openness to disclose research results and the use of a dedicated tooling platform. We use as an example an automated test generation approach that has been developed in the last two years collaboratively with Bombardier Transportation AB in Sweden. △ Less

Submitted 10 March, 2015; v1 submitted 2 September, 2014; originally announced September 2014.