-
Deriving and Evaluating a Detailed Taxonomy of Game Bugs
Authors:
Nigar Azhar Butt,
Salman Sherin,
Muhammad Uzair Khan,
Atif Aftab Jilani,
Muhammad Zohaib Iqbal
Abstract:
Game development has become an extremely competitive multi-billion-dollar industry. Many games fail even after years of development efforts because of game-breaking bugs that disrupt the game-play and ruin the player experience. The goal of this work is to provide a bug taxonomy for games that will help game developers in develo** bug-resistant games, game testers in designing and executing faul…
▽ More
Game development has become an extremely competitive multi-billion-dollar industry. Many games fail even after years of development efforts because of game-breaking bugs that disrupt the game-play and ruin the player experience. The goal of this work is to provide a bug taxonomy for games that will help game developers in develo** bug-resistant games, game testers in designing and executing fault-finding test cases, and researchers in evaluating game testing approaches. For this purpose, we performed a Multivocal Literature Review (MLR) by analyzing 436 sources, out of which 189 (78 academic and 111 grey) sources reporting bugs encountered in the game development industry were selected for analysis. We validate the proposed taxonomy by conducting a survey involving different game industry practitioners. The MLR allowed us to finalize a detailed taxonomy of 63 game bug categories in end-user perspective including eight first-tier categories: Gaming Balance, Implementation Response, Network, Sound, Temporal, Unexpected Crash, Navigational, and Non-Temporal faults. We observed that manual approaches towards game testing are still widely used. Only one of the approaches targets sound bugs whereas game balancing and how to incorporate machine learning in game testing is trending in the recent literature. Most of the game testing techniques are specialized and dependent on specific platforms.
△ Less
Submitted 28 November, 2023;
originally announced November 2023.
-
Landsca** Systematic Map** Studies in Software Engineering: A Tertiary Study
Authors:
Muhammad Uzair khan,
Salman Sherin,
Muhammad Zohaib Iqbal,
Rubab Zahid
Abstract:
Context: A number of Systematic Map** Studies (SMSs) that cover Software Engineering (SE) are reported in literature. Tertiary studies synthesize the secondary studies to provide a holistic view of an area. Objectives: We synthesize SMSs in SE to provide insights into existing SE areas and to investigate the trends and quality of SMSs. Methodology: We use Systematic Literature Review protocol to…
▽ More
Context: A number of Systematic Map** Studies (SMSs) that cover Software Engineering (SE) are reported in literature. Tertiary studies synthesize the secondary studies to provide a holistic view of an area. Objectives: We synthesize SMSs in SE to provide insights into existing SE areas and to investigate the trends and quality of SMSs. Methodology: We use Systematic Literature Review protocol to analyze and map the SMSs in SE, till August 2017, to SE Body of Knowledge (SWEBOK). Results: We analyze 210 SMSs and results show that: (1) Software design and construction are most active areas in SE; (2) Some areas lack SMSs, including mathematical foundations, software configuration management, and SE tools; (3) The quality of SMSs is improving with time; (4) SMSs in journals have higher quality than SMSs in conferences and are cited more often; (5) Low quality in SMSs can be attributed to a lack of quality assessment in SMSs and not reporting information about the primary studies. Conclusion: There is a potential for more SMSs in some SE areas. A number of SMSs do not provide the required information for an SMS, which leads to a low quality score.
△ Less
Submitted 24 September, 2019;
originally announced September 2019.
-
A Systematic Literature Review of Test Breakage Prevention and Repair Techniques
Authors:
Javaria Imtiaz,
Salman Sherin,
Muhammad Uzair khan,
Muhammad Zohaib Iqbal
Abstract:
Context: When an application evolves, some of the developed test cases break. Discarding broken test cases causes a significant waste of effort and leads to test suites that are less effective and have lower coverage. Test repair approaches evolve test suites along with applications by repairing the broken test cases. Objective: Numerous studies are published on test repair approaches every year.…
▽ More
Context: When an application evolves, some of the developed test cases break. Discarding broken test cases causes a significant waste of effort and leads to test suites that are less effective and have lower coverage. Test repair approaches evolve test suites along with applications by repairing the broken test cases. Objective: Numerous studies are published on test repair approaches every year. It is important to summarise and consolidate the existing knowledge in the area to provide directions to researchers and practitioners. This research work provides a systematic literature review in the area of test case repair and breakage prevention, aiming to guide researchers and practitioners in the field of software testing. Method: We followed the standard protocol for conducting a systematic literature review. First, research goals were defined using the Goal Question Metric (GQM). Then we formulate research questions corresponding to each goal. Finally, metrics are extracted from the included papers. Based on the defined selection criteria a final set of 41 primary studies are included for analysis. Results: The selection process resulted in 5 journal papers, and 36 conference papers. We present a taxonomy that lists the causes of test case breakages extracted from the literature. We found that only four proposed test repair tools are publicly available. Most studies evaluated their approaches on open-source case studies. Conclusion: There is significant room for future research on test repair techniques. Despite the positive trend of evaluating approaches on large scale open-source studies, there is a clear lack of results from studies done in a real industrial context. Few tools are publicly available which lowers the potential of adaption by industry practitioners.
△ Less
Submitted 24 September, 2019;
originally announced September 2019.
-
A Systematic Map** Study on Testing of Machine Learning Programs
Authors:
Salman Sherin,
Muhammad Uzair khan,
Muhammad Zohaib Iqbal
Abstract:
We aim to conduct a systematic map** in the area of testing ML programs. We identify, analyze and classify the existing literature to provide an overview of the area. We followed well-established guidelines of systematic map** to develop a systematic protocol to identify and review the existing literature. We formulate three sets of research questions, define inclusion and exclusion criteria a…
▽ More
We aim to conduct a systematic map** in the area of testing ML programs. We identify, analyze and classify the existing literature to provide an overview of the area. We followed well-established guidelines of systematic map** to develop a systematic protocol to identify and review the existing literature. We formulate three sets of research questions, define inclusion and exclusion criteria and systematically identify themes for the classification of existing techniques. We also report the quality of the published works using established assessment criteria. we finally selected 37 papers out of 1654 based on our selection criteria up to January 2019. We analyze trends such as contribution facet, research facet, test approach, type of ML and the kind of testing with several other attributes. We also discuss the empirical evidence and reporting quality of selected papers. The data from the study is made publicly available for other researchers and practitioners. We present an overview of the area by answering several research questions. The area is growing rapidly, however, there is lack of enough empirical evidence to compare and assess the effectiveness of the techniques. More publicly available tools are required for use of practitioners and researchers. Further attention is needed on non-functional testing and testing of ML programs using reinforcement learning. We believe that this study can help researchers and practitioners to obtain an overview of the area and identify several sub-areas where more research is required
△ Less
Submitted 11 July, 2019;
originally announced July 2019.
