Systematic Literature Review on Application of Learning-based Approaches in Continuous Integration ^††thanks: Citation: Arani, A. K., M., Le, T. H., M., Zahedi & Babar, M. A. (2023, May). Systematic Literature Review on Application of Learning-based Approaches in Continuous Integration. arXiv preprint (2024)

Our search string consists of two segments: A) “Machine Learning” and related synonyms, and B) “Continuous Integration”. These choices were informed by a review of previous studies and SLRs [1, 30, 31, 32], with the complete search string detailed in Table 2. The search string was executed on 26th July 2023, and snowballing was conducted on 7th August 2023.

Before implementing our search string to identify relevant literature, the first and third authors conducted a preliminary search on Google Scholar using keywords such as ‘machine learning,’ ‘Continuous Integration’, and related terms. This search aimed to identify initial relevant studies on the application of machine learning (ML) methods in Continuous Integration (CI). This preliminary search served as a validation set of papers. Subsequently, we iteratively refined our search string to retrieve all the papers included in our validation set. However, due to its tendency to yield numerous irrelevant results [1], Google Scholar was not used for our primary search.

To refine our search string, we iteratively ran it on ACM, IEEE, and Scopus databases, identifying and adding missing search terms for comprehensive coverage in the initial phase. The refined search string was then executed on these three databases, resulting in 1189 hits.

For collecting a list of high-quality studies, we established both inclusion and exclusion criteria. You can find the selection process and these criteria in Figure 2 and Table 3 respectively. We applied these criteria to all 1189 studies from the three databases, involving several steps in the paper selection process. Initially, non-English papers were excluded. Non-research documents, such as reports and notes, were disregarded. Additionally, papers with fewer than five pages were excluded for comprehensive method explanations. To prevent redundancy, duplicated papers across all databases were removed before reviewing the content.

In subsequent stages, the first and second authors reviewed paper titles and abstracts, categorizing them into two lists and curating a selection of potential papers relevant to ML in CI. A comprehensive examination of the full text followed, ensuring relevance and content quality, with the cooperation of the first and second authors and continuous input from the third and fourth authors. According to Wohlin’s guidelines [33], the selected papers must be published in peer-reviewed venues.

To ensure comprehensive coverage of relevant literature, we employed snowballing on the initial set of 50 papers [33], with the assistance of researchers whose names are presented in the acknowledgment section. Backward and forward snowballing involved reviewing references in selected papers and papers citing them, respectively. We repeated this process and reviewed steps until no more related papers could be identified.

In backward snowballing, 11 papers were identified, and in forward snowballing, four potential papers were found before full-text review. After reviewing these 15 potential papers, we filtered out 13 based on inclusion and exclusion criteria. In the end, 52 papers on the application of ML in CI were selected. The list and corresponding IDs are presented in the appendices, specifically in Table 22.

Since we ran the search string on July 26th 2023 and conducted the snowballing on August 7th 2023, we can say that a total of 52 studies were published between 2000 and August 2023. As shown in Figure 3, the first study on the application of ML techniques in CI was published in 2006 [36], which focused on predicting build validation in the CI phase. Notably, demographic data reveals that no paper was published on this topic between 2006 and 2014. However, the application of ML methods in CI has gradually gained more attention from researchers, and the number of studies started increasing since 2014. Additionally, the results indicate that a majority of papers (65.4%) were published at conferences. It is worth mentioning that from 2021 all the studies focused only on enhancing the regression testing and predicting the build validations. This point presents the importance of these two phases in the CI pipeline.

The concept of CI encompasses various areas and tools. To illustrate the areas that researchers have focused on, we can examine the keywords used in the selected studies. Figure 4 displays the frequency of each keyword in the primary studies and highlights the most frequent ones. The domains covered in this SLR are diverse, including Software Engineering concepts (e.g. Continuous Integration, Test Case Prioritization, Monitoring), software tools (e.g. Travis CI), and ML concepts (e.g. Classifier, Reward Function). The most common keywords in the selected studies are Continuous Integration, Machine Learning, Regression Testing, and Test Case Prioritization. This point depicts the importance of the regression testing phase of CI and especially its test case prioritization task.

Table 4 displays the distribution of published papers on the application of ML in CI across 35 venues, with 12 venues publishing more than one study. Among these 12 venues, IEEE/ACM International Conference on Automated Software Engineering (ASE), International Conference on Software Engineering (ICSE), International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (STA), and IEEE Transactions on Software Engineering (TSE) stand out as the top venues for publishing research on the topic of application of ML in CI, with each venue publishing three papers. It is worth noting that most of the papers (44.2%) were published in 23 different venues, with Software Engineering venues being responsible for the majority of papers (65.2%). The other domains of venues include Computer Science, Artificial Intelligence, Web and databases.

Table 5 provides a classification of the study context of the reviewed papers into three groups:“Industry”, “Open-Source”, and “Non-industry (simulation)”. Industrial studies were conducted with real-world data sets from software companies, such as projects in Microsoft company (S1) [37], to verify the applicability of the proposed methods in a practical closed-source software environment. On the other hand, studies in the “Open-Source” category validated their methods on real-world data projects from industrial open-source software, such as the Apache projects (S10) [11]. The distribution of the studies in Industrial and Open-Source categories presents the applicability of the ML methods in different CI environments.

Based on Table 5, it can be observed that only two studies (S12, S19) conducted research in simulated environments, while the majority of studies (96.2%) were situated in the Industry and Open-source categories. Simulation-based data sets refer to studies that implemented and evaluated their methods in simulated environments. In S19, the authors used the Gym library to simulate a CI environment by using execution logs of test cases and training a Reinforcement Learning (RL) agent. The simulator recreates a similar environment as a real environment for testing execution history. In S12, the authors simulated a cloud environment by installing packages manually and validated their ML method for discovering the installed software on containers and Virtual Machines (VMs) in testing the whole system. This highlights the usefulness of simulation-based data sets in evaluating and testing ML-based methods in a controlled and reproducible environment.

Moreover, we conducted a clustering analysis of the primary studies based on the types of research methods used. Typically, studies are categorized into two groups, namely quantitative and qualitative research, and we assigned the studies to each group based on their presented results and conclusions. Quantitative research methods involve the collection of empirical data through measurements and procedures, while qualitative studies are more descriptive and leave more room for interpretation [38].

In evaluating the effectiveness of quantitative research methods, researchers often rely on metrics such as the accuracy and performance of ML models, whereas qualitative methods may involve surveys, interviews, and other techniques to gather feedback from users of the trained ML models [39]. The results presented that none of the studies in this area employed qualitative methods for evaluating their solutions. Hence, a significant gap exists in the case of using qualitative analysis in the area of ML for CI. Furthermore, in the SLR paper, we provide additional information on the evaluation methods of quantitative research, which can assist researchers and practitioners in understanding how quantitative results can be validated and what are the most commonly used evaluation metrics in the area of ML for CI.

Summary: $\bullet$ The number of published studies related to the application of ML in CI has been on the rise, and gradually narrowed down to only regression testing and build validation over time. $\bullet$ All of the studies included in this SLR utilized quantitative research methods. However, a lack of qualitative assessment of the results is visible in the literature.

In the reviewed literature, 67 diverse data sources were utilized across the 52 selected ML-based CI studies. These sources differ in characteristics like lines of code (LOC), number of builds, and project specificity. For instance, study S3 used Google Dagger (848 LOC), contrasting with S2 employing a Samsung dataset with over 27 million LOC. Similarly, builds in different studies varied significantly, e.g., the Jazz project in S44 had 199 builds, while S5 used the TravisTorrent dataset with over 300,000 builds. These variations emphasize the need to comprehend data source characteristics in ML-based CI research.

To highlight frequently used sources, 11 publicly available datasets were employed in more than one study, as indicated by the numbers in parentheses.

To underscore the applicability of the data sources, we have presented 11 datasets that have been utilized in multiple studies and are reported as publicly available at the time of the studies’ publication. The number in parentheses indicates the frequency with which each dataset has been utilized in these studies.

IOF/ROL (12), Paint-Control (11) from ABB Robotics company, Apache-Commons (9), GSDTSR (9), Google Guava (5), Rails (5), TravisTorrent CI projects (5), MyBatis (4), Google Closure (3), Google Auto (3), DSpace (3), and Google Dagger (2).

Table 9 provides a summary of the properties of these datasets, with particular emphasis on datasets related to testing data in CI. Additionally, the data properties of Travis Torrent and Google Dagger are presented as N/A due to their unavailability at the time of writing this paper.

To assist researchers in selecting suitable datasets based on their defined research problems, this section presents key statistics and summaries of selected datasets. For instance, ABB company, a prominent industrial robot supplier, provides robot software and equipment, with datasets such as Paint Control (PC) and IOF/ROL containing historical information on test results and over 300 CI cycles. CI cycles encompass all development tasks, including coding, building, and testing, occurring continuously to ensure code integration and quality throughout the software development lifecycle before preparing a software product for deployment [50]. PC includes 89 test cases, 352 CI cycles, 25,594 verdicts, and a 19.36% failure rate, while IOF/ROL consists of 1941 test cases, 320 CI cycles, 32,260 verdicts, and a 28.79% failure rate. The GSDTSR dataset, an open resource from Google, has 336 CI cycles, with a notably low failure rate of around 0.25%. Considering these datasets alongside the previously discussed sources enables researchers to make informed choices based on their research objectives and requirements, facilitating evaluations in diverse dataset scenarios.

Table 9 highlights notable characteristics of datasets. The IOF/ROL and PC datasets stand out for their higher failure rates, a rarity in CI, making them popular among researchers. The Apache datasets gain popularity for their good structure and quality [51]. Additionally, the GSDTSR, Google Guava, Rails, MyBatis, and Google Closure datasets find frequent use due to their substantial test outcome volume, as indicated in Table 9.

TravisTorrent and Apache datasets are extensively employed in CI research. TravisTorrent, spanning 1,359 projects (402 Java, 898 Ruby, 59 in other languages), encompasses 2,640,825 builds. The Apache dataset comprises diverse projects like Cassandra, Ivy, Lang, Drill, and Math, featuring CI cycles ranging from 55 to 438. Researchers favor these datasets for their versatility, enabling evaluations across a broad spectrum of scenarios.

Table 9 illustrates the correlation between frequently used data sources and identified CI tasks in selected studies. Apache datasets prove versatile and employed across various CI phases due to project diversity. Conversely, TravisTorrent is notably applied in Build Prediction, and Google Dagger in predicting branch coverage during the Integration Test phase.

This section categorizes and explains data types within CI pipelines. Nine categories, including Source Code, Code Meta-Data (MD), Test Case, Test MD, Commit MD, Build Logs, Project MD, Textual Data or Texts, and System Logs, are identified. Table 10 provides a detailed overview of these data types, offering valuable insights for research purposes. Integrating innovative data types could unlock further exploration opportunities.

Here we aim to elucidate the correlation between data types and CI tasks, providing valuable insights into how different data types can enhance the efficiency of each task. This understanding informs the development and deployment of CI strategies, guiding the selection of data types for future research in automating CI tasks.

Table 11 highlights the prevalent use of source code metadata over raw source code data in the selected studies. This tendency is likely attributed to the growing availability of code analysis tools like the CK tool⁴⁴4https://github.com/mauricioaniche/ck, which automates the computation of established metrics such as Chidamber and Kemerer (CK). These tools have significantly streamlined the extraction of source code metadata for researchers. Notably, S3, S9, and S15 explicitly mentioned utilizing these tools for extracting CK metrics.

S15, S40, and S44 utilized Halstead metrics with available tools to delve into source code attributes, enriching the development of ML-based CI solutions. Extracting source code metadata from source code is cost-effective and straightforward, involving metrics like comments, methods, lines of code, and parameters. These metrics are easily obtainable and do not demand extensive computational resources. Moreover, these data types are well-suited for ML models as they can be readily converted into numerical values.

Table 11 reveals the prevalent use of source code metadata in studies related to Build Prediction, Test Optimization, and Defect Prediction in Regression Testing, as well as in Integration Testing within the CI pipeline. However, it is notably absent in the context of Unit Test Prediction, signifying a research gap in this area.

Table 11 underscores the limited utilization of data types such as test case codes, build logs, project metadata, textual data, and system logs in automating CI tasks. This underscores the potential benefits of exploring combinations of various data types for each CI task. For example, combining system logs with code metadata could enhance the accuracy of the Build Prediction task in CI. Furthermore, it is noteworthy that test metadata and commit metadata are predominantly employed in the Build Prediction task, indicating areas for further research in ML-based CI approaches.

In this section of our literature we mainly focus on data-related techniques used to modify and prepare raw data for ML models. Data cleaning, a fundamental aspect, was not explicitly examined separately due to limited information in the studies. However, data cleaning can be considered a filtering technique within the data preparation process.

Table 12 shows that the selected studies utilized ten distinct data preparation techniques. The choice of these techniques and the data’s characteristics depends on the overall strategy for addressing the research problem. The table summarizes the employed techniques and their subgroups. The next section provides a comprehensive description and examples for each data preparation group and subgroup.

DP1) Conditioning: This technique adjusts the dataset according to its characteristics and the proposed solution. In the case of manual data division, as demonstrated in study S5, the TravisTorrent dataset was partitioned using strategies such as the Burak filter [48]and the Bellwether filter [48]. In another instance (S21), data was segregated based on previous build results (pass or fail), resulting in separate training of machine learning models for each data partition.

Moreover, clustering methods like k-means can divide data based on feature values or data distribution, as in studies S24 and S45. Objective data selection involves choosing specific data points or segments based on defined objectives or criteria of our solution, such as selecting lines before and after the changed codes (changed codes are the objective of our solution) for inclusion in the dataset.

Utilizing conditioning for data preparation presents a clear advantage in reducing the computational overhead during ML model training, especially in CI environments with significant data volumes [42].

DP2) Building: This methodology transforms data to enhance its structure for better compatibility with ML techniques, especially in diverse CI environments. It consists of two sub-groups.

The first sub-group, data augmentation, is the most commonly used method. Researchers employ it to modify data structures or simulate real-world conditions. For instance, in S13, input data is padded with zeros for consistency and easier processing. Techniques like creating graphs or trees based on regression test results (S29, S36, S46) or simulating cloud-based real environments (S12) fall into this sub-group.

The second sub-group, data manipulation, involves methods such as text processing (S23) and merging distinct parts of a dataset by identifying correlations, such as combining issue tracking and VCS (S10).

DP3) Balancing: Many commonly used data sources display imbalances, which can substantially impact the performance and accuracy of classifiers [52]. Addressing this issue involves employing re-sampling techniques or training separate models for each class of input data. Re-sampling encompasses both oversampling, which involves increasing instances in the minority class (e.g., SMOTE), and under-sampling, as demonstrated in S30, which reduces the number of instances in the majority class. Additionally, combining both methods, as exemplified in SMOGN in S37 [53], presents another viable approach.

DP4) Filtering: Data pruning, a form of data cleaning, is vital in CI, addressing high computational workloads in large-scale datasets and improving the performance of the ML models [54]. Seven studies used this method, e.g., S11 removed unexecuted tests, S51 eliminated errored builds, and S9 removed incomplete test data. Issues like these arise from evolving software practices [55] and human errors [54]. Filtering is crucial in build validation studies, tackling interrupted builds [46]. Selective data filtering can enhance the quality of the training model and reduce training costs by prioritizing data segments that closely resemble the practical environment where the trained ML model will be deployed. For example, in study S24, TravisTorrent was constrained to Java-based projects employing Ant, Maven, and Gradle CI build tools, while Ruby-based projects were excluded.

Summary: $\bullet$ 12 datasets, nine data types, and ten processing techniques are identified in the reviewed studies. $\bullet$ Clustering datasets can reduce computation overhead and enhance ML model accuracy due to the high volume of CI data. $\bullet$ Addressing class imbalance is crucial to improve ML model accuracy in CI data. $\bullet$ Filtering, addressing exceptional cases like cancelled or errored processes, enhances ML model performance and reduces computational overhead.

Feature engineering is a crucial step in ML model development [26]. Its significance lies in two key factors:

Feature engineering techniques significantly impact the accuracy of trained ML models [62]. These techniques transform raw data into a format interpretable and meaningful for ML models [26]. In our analysis of the reviewed papers, we identified five groups of feature engineering (FE) techniques. However, it is worth noting that among the 52 studies in this systematic literature review, 18 did not provide details on their feature engineering techniques.

FE1) Imputation and Elimination: This technique addresses missing values in a dataset by replacing them with calculated or similar values, preventing data loss due to missing features [63]. In CI, it is particularly useful for handling exceptions. For example, S9 calculated the LOC difference when the “Number of Modified Lines” was unavailable. It can also derive values by considering interdependencies among data rows, such as finding the start and end times of test cases based on the log of previous execution times. Moreover, to reduce features and lower ML model training costs, elimination methods like the chi-square method can be used to remove closely related features. For instance, authors retained one of two similar features: the number of modified lines or added lines per data entry [53]. Given the large CI data volume, dimensional reduction techniques, as seen in S13, can further reduce computational overhead and input features in ML methods.

FE2) Feature Enhancement and Labeling: In CI environments, the presence of diverse data types and their high volume poses a challenge for data analysis. To tackle this challenge, various strategies are employed to enhance data comprehensibility for training ML models and improving their performance. In S10, a new labeling scheme categorized software management profile activities into H(high), M(edium), and L(ow). S46 categorized test cases as entirely or partially redundant based on coverage analysis, augmenting data with additional features. This methodology allows for enriching datasets with contextual information; for instance, S37 labeled commit dates to indicate holidays or regular days. Additionally, S16 introduced a new feature representing the percentage of detected failures relative to the total failures for each test case.

FE3) Outliers Handling: In CI environments, unexpected data may arise due to errors or non-repeatable situations, often attributed to human errors or specified limitations. For instance, test case durations may be constrained, and exceeding this time limit results in exceptional values, known as outliers. Outliers can be detected by defining thresholds or cutoff parameters, as demonstrated in S13 and S18.

FE4) Feature Scaling: Feature values exhibit diverse distributions and ranges, necessitating normalization through statistical methods. This technique rescales data to fit within a specific range (commonly [0, 1]) or standardizes values to the same magnitude using methods like log transformation, which mitigates the impact of extremely high or low values. For instance, S3 applied log transformation in conjunction with z-score, although they did not specify which feature required this transformation. Feature scaling is widely employed, with 20 out of 34 studies explaining their feature engineering techniques using this method, primarily due to the diversity of feature values and the prevalence of numerical features in CI environments.

FE5) Feature Tagging and Encoding: CI datasets often involve nominal values, such as test results like “pass”, “fail”, or “canceled”. To facilitate the training of ML models, input data must be made comprehensible, as these models rely on mathematical formulas [27]. One approach is to assign 0 and 1 values to class labels or represent feature values with binary strings or tags based on predefined rules [62, 64].

While encoding applies to various data types, it is particularly common in text-based datasets, involving the assignment of tokens or tags to different text segments. Effective tokenization requires a thorough understanding of the data and the selection of appropriate tokens for each distinct group or category. For example, in S13, authors defined specific tokens (tags) for various groups in source code, such as variables, operands, data types, spaces, and more.

In Table 15, we illustrate the relationship between feature engineering methods and the most commonly used datasets in the literature. It is noteworthy that certain datasets, in comparison with those presented in Table 9, are omitted here because the referenced studies did not report their feature engineering methods.

Summary: $\bullet$ Narrative, Numerical, Content, and Temporal feature types are used more frequently in the selected studies in comparison with other feature types. $\bullet$ Tools like IntelliJ Idea, RSA, and Aniche can extract features from code with low computational overhead. $\bullet$ New features can be defined through more complex computations and raw code analysis such as tagging or defining tree-based structures. $\bullet$ Feature engineering techniques like scaling, outlier handling, and elimination are still useful in CI environments. They cut computational load and boost ML model accuracy and performance.

Various ML algorithms, including supervised, unsupervised, semi-supervised, and reinforcement learning, play vital roles in ML-based approaches. Brief introductions to these methods are presented in the following.

Supervised Learning: This approach involves training a model on labelled data [17], requiring human effort or pre-existing data labelling before model training [9]. Supervised learning comprises two primary classes: classification, where the model categorizes outputs into fixed or discrete classes, and regression, where the model predicts continuous values [66].

Unsupervised learning: Unsupervised learning algorithms, unlike supervised learning, do not require data labeling before training. These models uncover data relationships and cluster data points [17]. Unsupervised clustering algorithms are particularly suited for large datasets where manual data labeling efforts are impractical [17]. Semi-supervised learning: Semi-supervised learning algorithms leverage labeled data to classify unlabeled data by identifying underlying data relationships [67], making them more practical when dealing with imprecise or noisy datasets containing both labeled and unlabeled data [68]. Lastly, Reinforcement Learning (RL) algorithms learn through trial and error, receiving rewards or penalties at each step until they achieve the desired output or accuracy [66]. RL algorithms benefit from continuous model updates, making them suitable for dynamically changing CI environments and data.

Among the primary studies, supervised learning is the most commonly employed approach, with 44 out of 52 studies utilizing it. Unsupervised learning is the second most used, with five out of 52 studies employing this approach, and one study using both supervised and unsupervised ML methods. Additionally, four studies applied semi-supervised learning algorithms, while 12 studies employed RL algorithms. Our observation also indicates a significant increase in the usage of RL algorithms in “Test Optimization” tasks since 2020. Notably, two out of the three state-of-the-art methods in this task (COLEMAN and RETECS, see Table 7) are RL-based methods, underscoring the growing applicability of RL-based algorithms in the realm of test optimization.

Our analysis also reveals that classification algorithms are the most widely employed methods in the CI context, with 32 out of 52 studies utilizing them. This preference can be attributed to the characteristics of testing, where ML-based binary classifiers excel in predicting test outcomes and build results without the need for explicit execution. Additionally, one study (S49) in the CI environment employed multi-class classification methods to categorize reported issues into five classes, aiding developers in identifying and addressing bug-related issues based on the most frequent labels in the training dataset.

Table 16 demonstrates that Decision Tree (DT) algorithms are the prevailing classification method in the CI environment, with 25 out of 52 studies employing them. The popularity of DT algorithms can be attributed to several factors, two of which are derived from the selected studies, while the third is drawn from the literature.

First, CI environments continuously generate vast amounts of data, and ML models require frequent updates [69, 70]. Training and updating DT algorithms demand low computational resources, making them a feasible choice in CI settings.

Second, DT algorithms have high performance in classifying unseen data [47].

Third, DT algorithms are interpretable and easily comprehensible for human users [71].

Table 16 shows that, except for one study (S12), RL algorithms have primarily been used in regression testing (RT) tasks within CI environments. In RT, a predefined set of test cases is established, and new test cases are incrementally added as new features are developed for the final software product. The objective of RT is to identify test cases with failure outcomes and prioritize their execution before those with passing outcomes. RL algorithms are well-suited for RT because this CI phase can be formulated as a sequential decision-making problem, a key characteristic of RL-based solutions [44]. Moreover, RL algorithms adapt to new data more effectively with frequent updates compared to other supervised and unsupervised ML algorithms [62].

However, researchers and practitioners should consider the advantages and disadvantages of RL-based models. For instance, the ROCKET solution [72] faces scalability issues in long runtime, while the RETECS method [73], an RL-based solution, requires a substantial amount of time for training.

Alongside the widespread adoption of DT and RL algorithms in regression testing, Table 16 highlights the utilization of Neural Network (NN) algorithms in 14 studies. Remarkably, NN algorithms have been applied in 7 out of 10 CI tasks, underscoring their flexibility in addressing various challenges within CI. The appeal of NN algorithms, despite their need for significant computational resources during training, lies in their capacity to automatically extract features from datasets and their high predictive accuracy [70].

In general, ML models comprise a basic formula that requires configuration by identifying the optimal values for their hyper-parameters [65]. Selected studies present varying perspectives on hyperparameter tuning. Some emphasize the importance of hyperparameter tuning in achieving the optimal performance of trained models [43], considering skip** this process as a potential threat that can affect model accuracy [6, 45]. However, other studies, like [74], found that tuning hyperparameters did not significantly impact model accuracy. Additionally, Al-Sabbagh et al. [41] reported that automatic hyperparameter tuning tools can be time-consuming, prompting them to manually tune hyperparameters to save time.

It is worth noting that hyperparameter tuning can affect model training speed. For instance, adjusting the ‘training rate’ hyperparameter can increase model instability while reducing training time or decrease training speed while enhancing overall model performance stability [27]. Hence, our investigation delves into how the selected studies in the application of ML algorithms within CI environments performed hyperparameter tuning.

Based on the data extracted from the selected studies, we categorized the hyper-parameter tuning strategies (HT) into five groups.

HT1) The first group used searching methods to find the best hyperparameters. These studies employed methods such as Grid Search (GS), Bayesian Search (BS), and Genetic Algorithm (GA).

In GS, researchers fix the domain of hyperparameters, and the algorithm finds the best combination from these fixed values [75]. In BS, the next hyperparameter values are determined based on the evaluation results of the previous values, avoiding unnecessary evaluations [76]. In GA, hyperparameters are adjusted iteratively by evaluating the model’s performance and making slight changes [77].

In S38, a unique Genetic Algorithms-based hyperparameter tuning method is utilized. Random values were assigned to each hyperparameter, and a single-point crossover operator generated two new offspring in the Crossover step. The best solutions were retained using the elitism method and fitness function calculation. In the Mutation step, hyperparameter values were slightly modified, and the Crossover step was repeated. This process continued until it reached the predetermined stop** criteria. Table 17 details the hyperparameter tuning methods used for ML models, acknowledging that one tuning method might be applied to multiple ML models in studies utilizing more than one ML method.