Showing 1–28 of 28 results for author: Emmerich, M

Multicriteria Optimization and Decision Making: Principles, Algorithms and Case Studies

Authors: Michael Emmerich, André Deutz

Abstract: Real-world decision and optimization problems, often involve constraints and conflicting criteria. For example, choosing a travel method must balance speed, cost, environmental footprint, and convenience. Similarly, designing an industrial process must consider safety, environmental impact, and cost efficiency. Ideal solutions where all objectives are optimally met are rare; instead, we seek good… ▽ More Real-world decision and optimization problems, often involve constraints and conflicting criteria. For example, choosing a travel method must balance speed, cost, environmental footprint, and convenience. Similarly, designing an industrial process must consider safety, environmental impact, and cost efficiency. Ideal solutions where all objectives are optimally met are rare; instead, we seek good compromises and aim to avoid lose-lose scenarios. Multicriteria optimization offers computational techniques to compute Pareto optimal solutions, aiding decision analysis and decision making. This reader offers an introduction to this topic and has been developed on the basis of the revised edition of the reader for the MSc computer science course "Multicriteria Optimization and Decision Analysis" at the Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands. This course was taught annually by the first author from 2007 to 2023 as a single semester course with lectures and practicals. Our aim was to make the material accessible to MSc students who do not study mathematics as their core discipline by introducing basic numerical analysis concepts when necessary and providing numerical examples for interesting cases. The introduction is organized in a unique didactic manner developed by the authors, starting from more simple concepts such as linear programming and single-point methods, and advancing from these to more difficult concepts such as optimality conditions for nonlinear optimization and set-oriented solution algorithms. Besides, we focus on the mathematical modeling and foundations rather than on specific algorithms, though not excluding the discussion of some representative examples of solution algorithms. △ Less

Submitted 2 July, 2024; v1 submitted 29 June, 2024; originally announced July 2024.

Comments: 102 pages, Lecture notes

MSC Class: 90C29 ACM Class: G.1.6

Modular and Extendable 1D-Simulation for Microfluidic Devices

Authors: Maria Emmerich, Florina Costamoling, Robert Wille

Abstract: Microfluidic devices have been the subject of considerable attention in recent years. The development of novel microfluidic devices, their evaluation, and their validation requires simulations. While common methods based on Computational Fluid Dynamics (CFD) can be time-consuming, 1D simulation provides an appealing alternative that leads to efficient results with reasonable quality. Current 1D si… ▽ More Microfluidic devices have been the subject of considerable attention in recent years. The development of novel microfluidic devices, their evaluation, and their validation requires simulations. While common methods based on Computational Fluid Dynamics (CFD) can be time-consuming, 1D simulation provides an appealing alternative that leads to efficient results with reasonable quality. Current 1D simulation tools cover specific microfluidic applications; however, these tools are still rare and not widely adopted. There is a need for a more versatile and adaptable tool that covers novel applications, like mixing and the addition of membranes, and allows easy extension, resulting in one comprehensive 1D simulation tool for microfluidic devices. In this work, we present an open-source, modular, and extendable 1D simulation approach for microfluidic devices, which is available as an open-source software package at https://github.com/cda-tum/mmft-modular-1D-simulator. To this end, we propose an implementation that consists of a base module (providing the core functionality) that can be extended with dedicated application-specific modules (providing dedicated support for common microfluidic applications such as mixing, droplets, membranes, etc.). Case studies show that this indeed allows to efficiently simulate a broad spectrum of microfluidic applications in a quality that matches previous results or even fabricated devices. △ Less

Submitted 21 June, 2024; originally announced June 2024.

The Hypervolume Indicator Hessian Matrix: Analytical Expression, Computational Time Complexity, and Sparsity

Authors: André H. Deutz, Michael T. M. Emmerich, Hao Wang

Abstract: The problem of approximating the Pareto front of a multiobjective optimization problem can be reformulated as the problem of finding a set that maximizes the hypervolume indicator. This paper establishes the analytical expression of the Hessian matrix of the map** from a (fixed size) collection of $n$ points in the $d$-dimensional decision space (or $m$ dimensional objective space) to the scalar… ▽ More The problem of approximating the Pareto front of a multiobjective optimization problem can be reformulated as the problem of finding a set that maximizes the hypervolume indicator. This paper establishes the analytical expression of the Hessian matrix of the map** from a (fixed size) collection of $n$ points in the $d$-dimensional decision space (or $m$ dimensional objective space) to the scalar hypervolume indicator value. To define the Hessian matrix, the input set is vectorized, and the matrix is derived by analytical differentiation of the map** from a vectorized set to the hypervolume indicator. The Hessian matrix plays a crucial role in second-order methods, such as the Newton-Raphson optimization method, and it can be used for the verification of local optimal sets. So far, the full analytical expression was only established and analyzed for the relatively simple bi-objective case. This paper will derive the full expression for arbitrary dimensions ($m\geq2$ objective functions). For the practically important three-dimensional case, we also provide an asymptotically efficient algorithm with time complexity in $O(n\log n)$ for the exact computation of the Hessian Matrix' non-zero entries. We establish a sharp bound of $12m-6$ for the number of non-zero entries. Also, for the general $m$-dimensional case, a compact recursive analytical expression is established, and its algorithmic implementation is discussed. Also, for the general case, some sparsity results can be established; these results are implied by the recursive expression. To validate and illustrate the analytically derived algorithms and results, we provide a few numerical examples using Python and Mathematica implementations. Open-source implementations of the algorithms and testing data are made available as a supplement to this paper. △ Less

Submitted 2 January, 2023; v1 submitted 8 November, 2022; originally announced November 2022.

Optimally Weighted Ensembles of Regression Models: Exact Weight Optimization and Applications

Authors: Patrick Echtenbruck, Martina Echtenbruck, Joost Batenburg, Thomas Bäck, Boris Naujoks, Michael Emmerich

Abstract: Automated model selection is often proposed to users to choose which machine learning model (or method) to apply to a given regression task. In this paper, we show that combining different regression models can yield better results than selecting a single ('best') regression model, and outline an efficient method that obtains optimally weighted convex linear combination from a heterogeneous set of… ▽ More Automated model selection is often proposed to users to choose which machine learning model (or method) to apply to a given regression task. In this paper, we show that combining different regression models can yield better results than selecting a single ('best') regression model, and outline an efficient method that obtains optimally weighted convex linear combination from a heterogeneous set of regression models. More specifically, in this paper, a heuristic weight optimization, used in a preceding conference paper, is replaced by an exact optimization algorithm using convex quadratic programming. We prove convexity of the quadratic programming formulation for the straightforward formulation and for a formulation with weighted data points. The novel weight optimization is not only (more) exact but also more efficient. The methods we develop in this paper are implemented and made available via github-open source. They can be executed on commonly available hardware and offer a transparent and easy to interpret interface. The results indicate that the approach outperforms model selection methods on a range of data sets, including data sets with mixed variable type from drug discovery applications. △ Less

Submitted 22 June, 2022; originally announced June 2022.

MSC Class: 62J02 ACM Class: I.2.6; I.2

Efficient Stochastic Simulation of Network Topology Effects on the Peak Number of Infections in Epidemic Outbreaks

Authors: Yulian Kuryliak, Michael Emmerich, Dmytro Dosyn

Abstract: This paper investigates the effect of the structure of the contact network on the dynamics of the epidemic outbreak. In particular, we focus on the peak number of critically infected nodes (PCIN), determining the maximum workload of intensive healthcare units which should be kept low. As a model and simulation method, we develop a continuous-time Markov chain (CTMC) model and an efficient simulati… ▽ More This paper investigates the effect of the structure of the contact network on the dynamics of the epidemic outbreak. In particular, we focus on the peak number of critically infected nodes (PCIN), determining the maximum workload of intensive healthcare units which should be kept low. As a model and simulation method, we develop a continuous-time Markov chain (CTMC) model and an efficient simulation-based on Gillespie's Stochastic Simulation Algorithm (SSA). This methods combine a realistic approximation of the stochastic process not relying on the assumptions of mean-field models and large asymptotically large population sizes as in differential equation models, and at the same time an efficient way to simulate networks of moderate size. The approach is analysed for different scenarios, based on data from the COVID-19 outbreak and demographic data from Ukraine. From these results we extract network topology features that need to be considered to effectively decrease the peak number of infections. The CTMC simulation is implemented in python and integrated in a dashboard that can be used for interactive exploration and it is made openly available. △ Less

Submitted 27 February, 2022; originally announced February 2022.

MSC Class: 60J74 ACM Class: I.6.0

Automatic Preference Based Multi-objective Evolutionary Algorithm on Vehicle Fleet Maintenance Scheduling Optimization

Authors: Yali Wang, Steffen Limmer, Markus Olhofer, Michael Emmerich, Thomas Baeck

Abstract: A preference based multi-objective evolutionary algorithm is proposed for generating solutions in an automatically detected knee point region. It is named Automatic Preference based DI-MOEA (AP-DI-MOEA) where DI-MOEA stands for Diversity-Indicator based Multi-Objective Evolutionary Algorithm). AP-DI-MOEA has two main characteristics: firstly, it generates the preference region automatically during… ▽ More A preference based multi-objective evolutionary algorithm is proposed for generating solutions in an automatically detected knee point region. It is named Automatic Preference based DI-MOEA (AP-DI-MOEA) where DI-MOEA stands for Diversity-Indicator based Multi-Objective Evolutionary Algorithm). AP-DI-MOEA has two main characteristics: firstly, it generates the preference region automatically during the optimization; secondly, it concentrates the solution set in this preference region. Moreover, the real-world vehicle fleet maintenance scheduling optimization (VFMSO) problem is formulated, and a customized multi-objective evolutionary algorithm (MOEA) is proposed to optimize maintenance schedules of vehicle fleets based on the predicted failure distribution of the components of cars. Furthermore, the customized MOEA for VFMSO is combined with AP-DI-MOEA to find maintenance schedules in the automatically generated preference region. Experimental results on multi-objective benchmark problems and our three-objective real-world application problems show that the newly proposed algorithm can generate the preference region accurately and that it can obtain better solutions in the preference region. Especially, in many cases, under the same budget, the Pareto optimal solutions obtained by AP-DI-MOEA dominate solutions obtained by MOEAs that pursue the entire Pareto front. △ Less

Submitted 23 January, 2021; originally announced January 2021.

Multiple Node Immunisation for Preventing Epidemics on Networks by Exact Multiobjective Optimisation of Cost and Shield-Value

Authors: Michael Emmerich, Joost Nibbeling, Marios Kefalas, Aske Plaat

Abstract: The general problem in this paper is vertex (node) subset selection with the goal to contain an infection that spreads in a network. Instead of selecting the single most important node, this paper deals with the problem of selecting multiple nodes for removal. As compared to previous work on multiple-node selection, the trade-off between cost and benefit is considered. The benefit is measured in t… ▽ More The general problem in this paper is vertex (node) subset selection with the goal to contain an infection that spreads in a network. Instead of selecting the single most important node, this paper deals with the problem of selecting multiple nodes for removal. As compared to previous work on multiple-node selection, the trade-off between cost and benefit is considered. The benefit is measured in terms of increasing the epidemic threshold which is a measure of how difficult it is for an infection to spread in a network. The cost is measured in terms of the number and size of nodes to be removed or controlled. Already in its single-objective instance with a fixed number of $k$ nodes to be removed, the multiple vertex immunisation problems have been proven to be NP-hard. Several heuristics have been developed to approximate the problem. In this work, we compare meta-heuristic techniques with exact methods on the Shield-value, which is a sub-modular proxy for the maximal eigenvalue and used in the current state-of-the-art greedy node-removal strategies. We generalise it to the multi-objective case and replace the greedy algorithm by a quadratic program (QP), which then can be solved with exact QP solvers. The main contribution of this paper is the insight that, if time permits, exact and problem-specific methods approximation should be used, which are often far better than Pareto front approximations obtained by general meta-heuristics. Based on these, it will be more effective to develop strategies for controlling real-world networks when the goal is to prevent or contain epidemic outbreaks. This paper is supported by ready to use Python implementation of the optimization methods and datasets. △ Less

Submitted 13 October, 2020; originally announced October 2020.

Comments: Based on the Master Thesis of Joost Nibbeling, LIACS, Leiden University, The Netherlands

MSC Class: 90C27; 05C69 ACM Class: J.3; G.2.2; G.4; I.2.8

Tackling Morpion Solitaire with AlphaZero-likeRanked Reward Reinforcement Learning

Authors: Hui Wang, Mike Preuss, Michael Emmerich, Aske Plaat

Abstract: Morpion Solitaire is a popular single player game, performed with paper and pencil. Due to its large state space (on the order of the game of Go) traditional search algorithms, such as MCTS, have not been able to find good solutions. A later algorithm, Nested Rollout Policy Adaptation, was able to find a new record of 82 steps, albeit with large computational resources. After achieving this record… ▽ More Morpion Solitaire is a popular single player game, performed with paper and pencil. Due to its large state space (on the order of the game of Go) traditional search algorithms, such as MCTS, have not been able to find good solutions. A later algorithm, Nested Rollout Policy Adaptation, was able to find a new record of 82 steps, albeit with large computational resources. After achieving this record, to the best of our knowledge, there has been no further progress reported, for about a decade. In this paper we take the recent impressive performance of deep self-learning reinforcement learning approaches from AlphaGo/AlphaZero as inspiration to design a searcher for Morpion Solitaire. A challenge of Morpion Solitaire is that the state space is sparse, there are few win/loss signals. Instead, we use an approach known as ranked reward to create a reinforcement learning self-play framework for Morpion Solitaire. This enables us to find medium-quality solutions with reasonable computational effort. Our record is a 67 steps solution, which is very close to the human best (68) without any other adaptation to the problem than using ranked reward. We list many further avenues for potential improvement. △ Less

Submitted 14 June, 2020; originally announced June 2020.

Comments: 4 pages, 2 figures. the first/ongoing attempt to tackle Morpion Solitaire using ranked reward reinforcement learning. submitted to SYNASC2020

A Novel Column Generation Heuristic for Airline Crew Pairing Optimization with Large-scale Complex Flight Networks

Authors: Divyam Aggarwal, Dhish Kumar Saxena, Saaju Pualose, Thomas Bäck, Michael Emmerich

Abstract: Crew Pairing Optimization (CPO) is critical for an airlines' business viability, given that the crew operating cost is second only to the fuel cost. CPO aims at generating a set of flight sequences (crew pairings) to cover all scheduled flights, at minimum cost, while satisfying several legality constraints. The state-of-the-art heavily relies on relaxing the underlying Integer Programming Problem… ▽ More Crew Pairing Optimization (CPO) is critical for an airlines' business viability, given that the crew operating cost is second only to the fuel cost. CPO aims at generating a set of flight sequences (crew pairings) to cover all scheduled flights, at minimum cost, while satisfying several legality constraints. The state-of-the-art heavily relies on relaxing the underlying Integer Programming Problem into a Linear Programming Problem, which in turn is solved through the Column Generation (CG) technique. However, with the alarmingly expanding airlines' operations, CPO is marred by the curse of dimensionality, rendering the exact CG-implementations obsolete, and necessitating the heuristic-based CG-implementations. Yet, in literature, the much prevalent large-scale complex flight networks involving multiple { crew bases and/or hub-and-spoke sub-networks, largely remain uninvestigated. This paper proposes a novel CG heuristic, which has enabled the in-house development of an Airline Crew Pairing Optimizer (AirCROP). The efficacy of the heuristic/AirCROP has been tested on real-world, large-scale, complex network instances with over 4,200 flights, 15 crew bases, and multiple hub-and-spoke sub-networks (resulting in billion-plus possible pairings). Notably, this paper has a dedicated focus on the proposed CG heuristic (not the entire AirCROP framework) based on balancing random exploration of pairings; exploitation of domain knowledge (on optimal solution features); and utilization of the past computational & search effort through archiving. Though this paper has an airline context, the proposed CG heuristic may find wider applications across different domains, by serving as a template on how to utilize domain knowledge to better tackle combinatorial optimization problems. △ Less

Submitted 2 July, 2021; v1 submitted 18 May, 2020; originally announced May 2020.

Comments: 26 pages, 8 figures, 6 tables, 5 Algorithms

Improving Many-Objective Evolutionary Algorithms by Means of Edge-Rotated Cones

Authors: Yali Wang, André Deutz, Thomas Bäck, Michael Emmerich

Abstract: Given a point in $m$-dimensional objective space, any $\varepsilon$-ball of a point can be partitioned into the incomparable, the dominated and dominating region. The ratio between the size of the incomparable region, and the dominated (and dominating) region decreases proportionally to $1/2^{m-1}$, i.e., the volume of the Pareto dominating orthant as compared to all other volumes. Due to this rea… ▽ More Given a point in $m$-dimensional objective space, any $\varepsilon$-ball of a point can be partitioned into the incomparable, the dominated and dominating region. The ratio between the size of the incomparable region, and the dominated (and dominating) region decreases proportionally to $1/2^{m-1}$, i.e., the volume of the Pareto dominating orthant as compared to all other volumes. Due to this reason, it gets increasingly unlikely that dominating points can be found by random, isotropic mutations. As a remedy to stagnation of search in many objective optimization, in this paper, we suggest to enhance the Pareto dominance order by involving an obtuse convex dominance cone in the convergence phase of an evolutionary optimization algorithm. We propose edge-rotated cones as generalizations of Pareto dominance cones for which the opening angle can be controlled by a single parameter only. The approach is integrated in several state-of-the-art multi-objective evolutionary algorithms (MOEAs) and tested on benchmark problems with four, five, six and eight objectives. Computational experiments demonstrate the ability of these edge-rotated cones to improve the performance of MOEAs on many-objective optimization problems. △ Less

Submitted 19 June, 2020; v1 submitted 15 April, 2020; originally announced April 2020.

A Tailored NSGA-III Instantiation for Flexible Job Shop Scheduling

Authors: Yali Wang, Bas van Stein, Michael T. M. Emmerich, Thomas Bäck

Abstract: A customized multi-objective evolutionary algorithm (MOEA) is proposed for the multi-objective flexible job shop scheduling problem (FJSP). It uses smart initialization approaches to enrich the first generated population, and proposes various crossover operators to create a better diversity of offspring. Especially, the MIP-EGO configurator, which can tune algorithm parameters, is adopted to autom… ▽ More A customized multi-objective evolutionary algorithm (MOEA) is proposed for the multi-objective flexible job shop scheduling problem (FJSP). It uses smart initialization approaches to enrich the first generated population, and proposes various crossover operators to create a better diversity of offspring. Especially, the MIP-EGO configurator, which can tune algorithm parameters, is adopted to automatically tune operator probabilities. Furthermore, different local search strategies are employed to explore the neighborhood for better solutions. In general, the algorithm enhancement strategy can be integrated with any standard EMO algorithm. In this paper, it has been combined with NSGA-III to solve benchmark multi-objective FJSPs, whereas an off-the-shelf implementation of NSGA-III is not capable of solving the FJSP. The experimental results show excellent performance with less computing budget. △ Less

Submitted 14 April, 2020; originally announced April 2020.

On Initializing Airline Crew Pairing Optimization for Large-scale Complex Flight Networks

Authors: Divyam Aggarwal, Dhish Kumar Saxena, Thomas Bäck, Michael Emmerich

Abstract: Crew pairing optimization (CPO) is critically important for any airline, since its crew operating costs are second-largest, next to the fuel-cost. CPO aims at generating a set of flight sequences (crew pairings) covering a flight-schedule, at minimum-cost, while satisfying several legality constraints. For large-scale complex flight networks, billion-plus legal pairings (variables) are possible, r… ▽ More Crew pairing optimization (CPO) is critically important for any airline, since its crew operating costs are second-largest, next to the fuel-cost. CPO aims at generating a set of flight sequences (crew pairings) covering a flight-schedule, at minimum-cost, while satisfying several legality constraints. For large-scale complex flight networks, billion-plus legal pairings (variables) are possible, rendering their offline enumeration intractable and an exhaustive search for their minimum-cost full flight-coverage subset impractical. Even generating an initial feasible solution (IFS: a manageable set of legal pairings covering all flights), which could be subsequently optimized is a difficult (NP-complete) problem. Though, as part of a larger project the authors have developed a crew pairing optimizer (AirCROP), this paper dedicatedly focuses on IFS-generation through a novel heuristic based on divide-and-cover strategy and Integer Programming. For real-world large and complex flight network datasets (including over 3200 flights and 15 crew bases) provided by GE Aviation, the proposed heuristic shows upto a ten-fold speed improvement over another state-of-the-art approach. Unprecedentedly, this paper presents an empirical investigation of the impact of IFS-cost on the final (optimized) solution-cost, revealing that too low an IFS-cost does not necessarily imply faster convergence for AirCROP or even lower cost for the optimized solution. △ Less

Submitted 15 March, 2020; originally announced March 2020.

Comments: 17 pages, 9 figures, manuscript submitted for review in a refereed journal

Analysis of Hyper-Parameters for Small Games: Iterations or Epochs in Self-Play?

Authors: Hui Wang, Michael Emmerich, Mike Preuss, Aske Plaat

Abstract: The landmark achievements of AlphaGo Zero have created great research interest into self-play in reinforcement learning. In self-play, Monte Carlo Tree Search is used to train a deep neural network, that is then used in tree searches. Training itself is governed by many hyperparameters.There has been surprisingly little research on design choices for hyper-parameter values and loss-functions, pres… ▽ More The landmark achievements of AlphaGo Zero have created great research interest into self-play in reinforcement learning. In self-play, Monte Carlo Tree Search is used to train a deep neural network, that is then used in tree searches. Training itself is governed by many hyperparameters.There has been surprisingly little research on design choices for hyper-parameter values and loss-functions, presumably because of the prohibitive computational cost to explore the parameter space. In this paper, we investigate 12 hyper-parameters in an AlphaZero-like self-play algorithm and evaluate how these parameters contribute to training. We use small games, to achieve meaningful exploration with moderate computational effort. The experimental results show that training is highly sensitive to hyper-parameter choices. Through multi-objective analysis we identify 4 important hyper-parameters to further assess. To start, we find surprising results where too much training can sometimes lead to lower performance. Our main result is that the number of self-play iterations subsumes MCTS-search simulations, game-episodes, and training epochs. The intuition is that these three increase together as self-play iterations increase, and that increasing them individually is sub-optimal. A consequence of our experiments is a direct recommendation for setting hyper-parameter values in self-play: the overarching outer-loop of self-play iterations should be maximized, in favor of the three inner-loop hyper-parameters, which should be set at lower values. A secondary result of our experiments concerns the choice of optimization goals, for which we also provide recommendations. △ Less

Submitted 12 March, 2020; originally announced March 2020.

Airline Crew Pairing Optimization Framework for Large Networks with Multiple Crew Bases and Hub-and-Spoke Subnetworks

Authors: Divyam Aggarwal, Dhish Kumar Saxena, Thomas Bäck, Michael Emmerich

Abstract: Crew Pairing Optimization aims at generating a set of flight sequences (crew pairings), covering all flights in an airline's flight schedule, at minimum cost, while satisfying several legality constraints. CPO is critically important for airlines' business viability, considering that the crew operating cost is their second-largest expense. It poses an NP-hard combinatorial optimization problem, to… ▽ More Crew Pairing Optimization aims at generating a set of flight sequences (crew pairings), covering all flights in an airline's flight schedule, at minimum cost, while satisfying several legality constraints. CPO is critically important for airlines' business viability, considering that the crew operating cost is their second-largest expense. It poses an NP-hard combinatorial optimization problem, to tackle which, the state-of-the-art relies on relaxing the underlying Integer Programming Problem (IPP) into a Linear Programming Problem (LPP), solving the latter through Column Generation (CG) technique, and integerization of the resulting LPP solution. However, with the growing scale and complexity of the flight networks (those with a large number of flights, multiple crew bases and/or multiple hub-and-spoke subnetworks), the utility of the conventional CG-practices has become questionable. This paper proposed an Airline Crew Pairing Optimization Framework, AirCROP, whose constitutive modules include the Legal Crew Pairing Generator, Initial Feasible Solution Generator, and an Optimization Engine built on heuristic-based CG-implementation. In this paper, besides the design of AirCROP's modules, insights into important questions related to how these modules interact, which the literature is otherwise silent on, have been shared. These relate to the sensitivity of AirCROP's performance towards: sources of variability over multiple runs for a given problem, initialization method, and termination parameters for LPP-solutioning and IPP-solutioning. The efficacy of the AirCROP has been demonstrated on real-world large-scale and complex flight networks (with over 4200 flights, 15 crew bases, and billion-plus pairings). It is hoped that with the emergence of such complex flight networks, this paper shall serve as an important milestone for affiliated research and applications. △ Less

Submitted 18 November, 2020; v1 submitted 9 March, 2020; originally announced March 2020.

Comments: 28 pages, 3 figures, 9 tables, manuscript submitted for review in a refereed journal. A patent application, based on this research, has been filed in the Netherlands Patent Office. Moreover, D. Aggarwal (author) received the IEEE-ITSS Young Professionals Travelling Fellowship Award for presenting this research work at IEEE ITSC 2019, held in Auckland, New Zealand in October 2019

Real-World Airline Crew Pairing Optimization: Customized Genetic Algorithm versus Column Generation Method

Authors: Divyam Aggarwal, Dhish Kumar Saxena, Thomas Back, Michael Emmerich

Abstract: Airline crew pairing optimization problem (CPOP) aims to find a set of flight sequences (crew pairings) that cover all flights in an airline's highly constrained flight schedule at minimum cost. Since crew cost is second only to the fuel cost, CPOP solutioning is critically important for an airline. However, CPOP is NP-hard, and tackling it is quite challenging. The literature suggests, that when… ▽ More Airline crew pairing optimization problem (CPOP) aims to find a set of flight sequences (crew pairings) that cover all flights in an airline's highly constrained flight schedule at minimum cost. Since crew cost is second only to the fuel cost, CPOP solutioning is critically important for an airline. However, CPOP is NP-hard, and tackling it is quite challenging. The literature suggests, that when the CPOP's scale and complexity is reasonably limited, and an enumeration of all crew pairings is possible, then Metaheuristics are used, predominantly Genetic Algorithms (GAs). Else, Column Generation (CG) based Mixed Integer Programming techniques are used. Notably, as per the literature, a maximum of 45,000 crew pairings have been tackled by GAs. In a significant departure, this paper considers over 800 flights of a US-based large airline (with a monthly network of over 33,000 flights), and tests the efficacy of GAs by enumerating all 400,000+ crew pairings, apriori. Towards it, this paper proposes a domain-knowledge-driven customized-GA. The utility of incorporating domain-knowledge in GA operations, particularly initialization and crossover, is highlighted through suitable experiments. Finally, the proposed GA's performance is compared with a CG-based approach (developed in-house by the authors). Though the latter is found to perform better in terms of solution's cost-quality and run time, it is hoped that this paper will help in better understanding the strengths and limitations of domain-knowledge-driven customizations in GAs, for solving combinatorial optimization problems, including CPOPs. △ Less

Submitted 27 May, 2023; v1 submitted 8 March, 2020; originally announced March 2020.

Comments: 14 pages, 3 figures, 5 tables

Journal ref: In: Evolutionary Multi-Criterion Optimization. EMO 2023. Lecture Notes in Computer Science, vol 13970. Springer, Cham

Efficient Computation of Expected Hypervolume Improvement Using Box Decomposition Algorithms

Authors: Kaifeng Yang, Michael Emmerich, André Deutz, Thomas Bäck

Abstract: In the field of multi-objective optimization algorithms, multi-objective Bayesian Global Optimization (MOBGO) is an important branch, in addition to evolutionary multi-objective optimization algorithms (EMOAs). MOBGO utilizes Gaussian Process models learned from previous objective function evaluations to decide the next evaluation site by maximizing or minimizing an infill criterion. A common crit… ▽ More In the field of multi-objective optimization algorithms, multi-objective Bayesian Global Optimization (MOBGO) is an important branch, in addition to evolutionary multi-objective optimization algorithms (EMOAs). MOBGO utilizes Gaussian Process models learned from previous objective function evaluations to decide the next evaluation site by maximizing or minimizing an infill criterion. A common criterion in MOBGO is the Expected Hypervolume Improvement (EHVI), which shows a good performance on a wide range of problems, with respect to exploration and exploitation. However, so far it has been a challenge to calculate exact EHVI values efficiently. In this paper, an efficient algorithm for the computation of the exact EHVI for a generic case is proposed. This efficient algorithm is based on partitioning the integration volume into a set of axis-parallel slices. Theoretically, the upper bound time complexities are improved from previously $O (n^2)$ and $O(n^3)$, for two- and three-objective problems respectively, to $Θ(n\log n)$, which is asymptotically optimal. This article generalizes the scheme in higher dimensional case by utilizing a new hyperbox decomposition technique, which was proposed by D{ä}chert et al, EJOR, 2017. It also utilizes a generalization of the multilayered integration scheme that scales linearly in the number of hyperboxes of the decomposition. The speed comparison shows that the proposed algorithm in this paper significantly reduces computation time. Finally, this decomposition technique is applied in the calculation of the Probability of Improvement (PoI). △ Less

Submitted 13 June, 2019; v1 submitted 26 April, 2019; originally announced April 2019.

Hyper-Parameter Sweep on AlphaZero General

Authors: Hui Wang, Michael Emmerich, Mike Preuss, Aske Plaat

Abstract: Since AlphaGo and AlphaGo Zero have achieved breakground successes in the game of Go, the programs have been generalized to solve other tasks. Subsequently, AlphaZero was developed to play Go, Chess and Shogi. In the literature, the algorithms are explained well. However, AlphaZero contains many parameters, and for neither AlphaGo, AlphaGo Zero nor AlphaZero, there is sufficient discussion about h… ▽ More Since AlphaGo and AlphaGo Zero have achieved breakground successes in the game of Go, the programs have been generalized to solve other tasks. Subsequently, AlphaZero was developed to play Go, Chess and Shogi. In the literature, the algorithms are explained well. However, AlphaZero contains many parameters, and for neither AlphaGo, AlphaGo Zero nor AlphaZero, there is sufficient discussion about how to set parameter values in these algorithms. Therefore, in this paper, we choose 12 parameters in AlphaZero and evaluate how these parameters contribute to training. We focus on three objectives~(training loss, time cost and playing strength). For each parameter, we train 3 models using 3 different values~(minimum value, default value, maximum value). We use the game of play 6$\times$6 Othello, on the AlphaZeroGeneral open source re-implementation of AlphaZero. Overall, experimental results show that different values can lead to different training results, proving the importance of such a parameter sweep. We categorize these 12 parameters into time-sensitive parameters and time-friendly parameters. Moreover, through multi-objective analysis, this paper provides an insightful basis for further hyper-parameter optimization. △ Less

Submitted 19 March, 2019; originally announced March 2019.

Comments: 19 pages 13 figures

Assessing the Potential of Classical Q-learning in General Game Playing

Authors: Hui Wang, Michael Emmerich, Aske Plaat

Abstract: After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) pro… ▽ More After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee $\&$ Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex)\footnote{source code: https://github.com/wh1992v/ggp-rl}, to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the $ε$-greedy strategy, we propose a first enhancement, the dynamic $ε$ algorithm. In addition, inspired by (Gelly $\&$ Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games. △ Less

Submitted 14 October, 2018; originally announced October 2018.

Comments: arXiv admin note: substantial text overlap with arXiv:1802.05944

B $\rightarrow f_2(1270)$ form factors with light-cone sum rules

Authors: Maximilian Emmerich, Matthias Strohmaier, Andreas Schäfer

Abstract: We construct the quark-antiquark chiral odd distribution amplitudes including twist-four mass contributions for tensor mesons. We also give quark-antiquark-gluon distribution amplitudes, where we calculate the input parameters with QCD sum rules. With the help of equations of motion we determine the twist-three and twist-four distribution amplitudes including SU(3) breaking terms. We use QCD light… ▽ More We construct the quark-antiquark chiral odd distribution amplitudes including twist-four mass contributions for tensor mesons. We also give quark-antiquark-gluon distribution amplitudes, where we calculate the input parameters with QCD sum rules. With the help of equations of motion we determine the twist-three and twist-four distribution amplitudes including SU(3) breaking terms. We use QCD light-come sum rules to derive the form factors for the decay B $\rightarrow f_2(1270)$ with vector, axial-vector and tensor currents. We also give the $q^2$ dependence of the form factors. △ Less

Submitted 9 April, 2018; originally announced April 2018.

Journal ref: Phys. Rev. D 98, 014008 (2018)

Maximum Volume Subset Selection for Anchored Boxes

Authors: Karl Bringmann, Sergio Cabello, Michael T. M. Emmerich

Abstract: Let $B$ be a set of $n$ axis-parallel boxes in $\mathbb{R}^d$ such that each box has a corner at the origin and the other corner in the positive quadrant of $\mathbb{R}^d$, and let $k$ be a positive integer. We study the problem of selecting $k$ boxes in $B$ that maximize the volume of the union of the selected boxes. This research is motivated by applications in skyline queries for databases and… ▽ More Let $B$ be a set of $n$ axis-parallel boxes in $\mathbb{R}^d$ such that each box has a corner at the origin and the other corner in the positive quadrant of $\mathbb{R}^d$, and let $k$ be a positive integer. We study the problem of selecting $k$ boxes in $B$ that maximize the volume of the union of the selected boxes. This research is motivated by applications in skyline queries for databases and in multicriteria optimization, where the problem is known as the hypervolume subset selection problem. It is known that the problem can be solved in polynomial time in the plane, while the best known running time in any dimension $d \ge 3$ is $Ω\big(\binom{n}{k}\big)$. We show that: - The problem is NP-hard already in 3 dimensions. - In 3 dimensions, we break the bound $Ω\big(\binom{n}{k}\big)$, by providing an $n^{O(\sqrt{k})}$ algorithm. - For any constant dimension $d$, we present an efficient polynomial-time approximation scheme. △ Less

Submitted 2 March, 2018; originally announced March 2018.

Comments: Presented at SoCG'17. Full Version. 24 pages

ACM Class: F.2.2

Monte Carlo Q-learning for General Game Playing

Authors: Hui Wang, Michael Emmerich, Aske Plaat

Abstract: After the recent groundbreaking results of AlphaGo, we have seen a strong interest in reinforcement learning in game playing. General Game Playing (GGP) provides a good testbed for reinforcement learning. In GGP, a specification of games rules is given. GGP problems can be solved by reinforcement learning. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Ban… ▽ More After the recent groundbreaking results of AlphaGo, we have seen a strong interest in reinforcement learning in game playing. General Game Playing (GGP) provides a good testbed for reinforcement learning. In GGP, a specification of games rules is given. GGP problems can be solved by reinforcement learning. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al. As expected, Q-learning converges, although much slower than MCTS. Borrowing an idea from MCTS, we enhance Q-learning with Monte Carlo Search, to give QM-learning. This enhancement improves the performance of pure Q-learning. We believe that QM-learning can also be used to improve performance of reinforcement learning further for larger games, something which we will test in future work. △ Less

Submitted 21 May, 2018; v1 submitted 16 February, 2018; originally announced February 2018.

Cluster-based Kriging Approximation Algorithms for Complexity Reduction

Authors: Bas van Stein, Hao Wang, Wojtek Kowalczyk, Michael Emmerich, Thomas Bäck

Abstract: Kriging or Gaussian Process Regression is applied in many fields as a non-linear regression model as well as a surrogate model in the field of evolutionary computation. However, the computational and space complexity of Kriging, that is cubic and quadratic in the number of data points respectively, becomes a major bottleneck with more and more data available nowadays. In this paper, we propose a g… ▽ More Kriging or Gaussian Process Regression is applied in many fields as a non-linear regression model as well as a surrogate model in the field of evolutionary computation. However, the computational and space complexity of Kriging, that is cubic and quadratic in the number of data points respectively, becomes a major bottleneck with more and more data available nowadays. In this paper, we propose a general methodology for the complexity reduction, called cluster Kriging, where the whole data set is partitioned into smaller clusters and multiple Kriging models are built on top of them. In addition, four Kriging approximation algorithms are proposed as candidate algorithms within the new framework. Each of these algorithms can be applied to much larger data sets while maintaining the advantages and power of Kriging. The proposed algorithms are explained in detail and compared empirically against a broad set of existing state-of-the-art Kriging approximation methods on a well-defined testing framework. According to the empirical study, the proposed algorithms consistently outperform the existing algorithms. Moreover, some practical suggestions are provided for using the proposed algorithms. △ Less

Submitted 4 February, 2017; originally announced February 2017.

Comments: Submitted to IEEE Computational Intelligence Magazine for review

An Ontology of Preference-Based Multiobjective Metaheuristics

Authors: Longmei Li, Iryna Yevseyeva, Vitor Basto-Fernandes, Heike Trautmann, Ning **g, Michael Emmerich

Abstract: User preference integration is of great importance in multi-objective optimization, in particular in many objective optimization. Preferences have long been considered in traditional multicriteria decision making (MCDM) which is based on mathematical programming. Recently, it is integrated in multi-objective metaheuristics (MOMH), resulting in focus on preferred parts of the Pareto front instead o… ▽ More User preference integration is of great importance in multi-objective optimization, in particular in many objective optimization. Preferences have long been considered in traditional multicriteria decision making (MCDM) which is based on mathematical programming. Recently, it is integrated in multi-objective metaheuristics (MOMH), resulting in focus on preferred parts of the Pareto front instead of the whole Pareto front. The number of publications on preference-based multi-objective metaheuristics has increased rapidly over the past decades. There already exist various preference handling methods and MOMH methods, which have been combined in diverse ways. This article proposes to use the Web Ontology Language (OWL) to model and systematize the results developed in this field. A review of the existing work is provided, based on which an ontology is built and instantiated with state-of-the-art results. The OWL ontology is made public and open to future extension. Moreover, the usage of the ontology is exemplified for different use-cases, including querying for methods that match an engineering application, bibliometric analysis, checking existence of combinations of preference models and MOMH techniques, and discovering opportunities for new research and open research questions. △ Less

Submitted 10 March, 2017; v1 submitted 26 September, 2016; originally announced September 2016.

Comments: submitted to European Journal of Operational Research

The decays $Λ_{b,c}\to N^*\, l\,ν$ in QCD

Authors: Maximilian Emmerich, Nils Offen, Andreas Schäfer

Abstract: We present an exploratory study of the $Λ_{c,b}\to N^*$-form factors and the semileptonic decay width within the framework of light-cone sum rules. We use two different methods and two different interpolating currents for the $Λ_{c,b}$. As interpolating currents we choose an axial-vector like and a pseudoscalar like current. Our results show that the procedure of eliminating negative parity partne… ▽ More We present an exploratory study of the $Λ_{c,b}\to N^*$-form factors and the semileptonic decay width within the framework of light-cone sum rules. We use two different methods and two different interpolating currents for the $Λ_{c,b}$. As interpolating currents we choose an axial-vector like and a pseudoscalar like current. Our results show that the procedure of eliminating negative parity partners is not well suited for the case at hand and that the second approach using structures with highest powers of $p_+$ with an axial-vector like interpolating current gives the most reliable results. Our predictions are based on the models obtained in \cite{Anikin:2015ita,Braun:2014wpa}. The largest uncertainty comes from the uncertainty of the twist 4 parameters $η_{10},\,η_{11}$ and we take the spread between the two models in \cite{Anikin:2015ita} as a measure for this. We get \begin{eqnarray} Γ(Λ_b\to N^*(1535) l ν)&=&\left(0.0058^{+0.0010}_{-0.0009}\right)\cdot\left(\dfrac{V_{ub}}{3.5\cdot10^{-3}}\right)^2,\quad\mbox{LCSR(1)} \nonumber Γ(Λ_b\to N^*(1535) l ν)&=&\left(0.00070^{+0.00012}_{-0.00011}\right)\cdot\left(\dfrac{V_{ub}}{3.5\cdot10^{-3}}\right)^2,\quad\mbox{LCSR(2)} \nonumber Γ(Λ_c\to N^*(1535) l ν)&=&\left(0.0064^{+0.0012}_{-0.0011}\right)\cdot \left(\dfrac{V_{cd}}{0.225}\right)^2,\quad\mbox{LCSR(1)}\nonumber Γ(Λ_c\to N^*(1535) l ν)&=&\left(0.00077^{+0.00016}_{-0.00014}\right)\cdot \left(\dfrac{V_{cd}}{0.225}\right)^2,\quad\mbox{LCSR(2)}\nonumber \end{eqnarray} as predictions for the respective decay widths, where LCSR(1) and LCSR(2) refer to the two different models of the distribution amplitudes of the $N^*$. It is seen that even a rough measurement of these decays will greatly help to discriminate different models. △ Less

Submitted 22 April, 2016; originally announced April 2016.

Comments: 15 pages, 6 figures

Solving the G-problems in less than 500 iterations: Improved efficient constrained optimization by surrogate modeling and adaptive parameter control

Authors: Samineh Bagheri, Wolfgang Konen, Michael Emmerich, Thomas Bäck

Abstract: Constrained optimization of high-dimensional numerical problems plays an important role in many scientific and industrial applications. Function evaluations in many industrial applications are severely limited and no analytical information about objective function and constraint functions is available. For such expensive black-box optimization tasks, the constraint optimization algorithm COBRA was… ▽ More Constrained optimization of high-dimensional numerical problems plays an important role in many scientific and industrial applications. Function evaluations in many industrial applications are severely limited and no analytical information about objective function and constraint functions is available. For such expensive black-box optimization tasks, the constraint optimization algorithm COBRA was proposed, making use of RBF surrogate modeling for both the objective and the constraint functions. COBRA has shown remarkable success in solving reliably complex benchmark problems in less than 500 function evaluations. Unfortunately, COBRA requires careful adjustment of parameters in order to do so. In this work we present a new self-adjusting algorithm SACOBRA, which is based on COBRA and capable to achieve high-quality results with very few function evaluations and no parameter tuning. It is shown with the help of performance profiles on a set of benchmark problems (G-problems, MOPTA08) that SACOBRA consistently outperforms any COBRA algorithm with fixed parameter setting. We analyze the importance of the several new elements in SACOBRA and find that each element of SACOBRA plays a role to boost up the overall optimization performance. We discuss the reasons behind and get in this way a better understanding of high-quality RBF surrogate modeling. △ Less

Submitted 31 December, 2015; originally announced December 2015.

Multiobjective Optimization of Classifiers by Means of 3-D Convex Hull Based Evolutionary Algorithm

Authors: Jiaqi Zhao, Vitor Basto Fernandes, Licheng Jiao, Iryna Yevseyeva, Asep Maulana, Rui Li, Thomas Bäck, Michael T. M. Emmerich

Abstract: Finding a good classifier is a multiobjective optimization problem with different error rates and the costs to be minimized. The receiver operating characteristic is widely used in the machine learning community to analyze the performance of parametric classifiers or sets of Pareto optimal classifiers. In order to directly compare two sets of classifiers the area (or volume) under the convex hull… ▽ More Finding a good classifier is a multiobjective optimization problem with different error rates and the costs to be minimized. The receiver operating characteristic is widely used in the machine learning community to analyze the performance of parametric classifiers or sets of Pareto optimal classifiers. In order to directly compare two sets of classifiers the area (or volume) under the convex hull can be used as a scalar indicator for the performance of a set of classifiers in receiver operating characteristic space. Recently, the convex hull based multiobjective genetic programming algorithm was proposed and successfully applied to maximize the convex hull area for binary classification problems. The contribution of this paper is to extend this algorithm for dealing with higher dimensional problem formulations. In particular, we discuss problems where parsimony (or classifier complexity) is stated as a third objective and multi-class classification with three different true classification rates to be maximized. The design of the algorithm proposed in this paper is inspired by indicator-based evolutionary algorithms, where first a performance indicator for a solution set is established and then a selection operator is designed that complies with the performance indicator. In this case, the performance indicator will be the volume under the convex hull. The algorithm is tested and analyzed in a proof of concept study on different benchmarks that are designed for measuring its capability to capture relevant parts of a convex hull. Further benchmark and application studies on email classification and feature selection round up the analysis and assess robustness and usefulness of the new algorithm in real world settings. △ Less

Submitted 17 December, 2014; originally announced December 2014.

Comments: 32 pages, 26 figures

Faster Computation of Expected Hypervolume Improvement

Authors: Iris Hupkens, Michael Emmerich, André Deutz

Abstract: The expected improvement algorithm (or efficient global optimization) aims for global continuous optimization with a limited budget of black-box function evaluations. It is based on a statistical model of the function learned from previous evaluations and an infill criterion - the expected improvement - used to find a promising point for a new evaluation. The `expected improvement' infill criterio… ▽ More The expected improvement algorithm (or efficient global optimization) aims for global continuous optimization with a limited budget of black-box function evaluations. It is based on a statistical model of the function learned from previous evaluations and an infill criterion - the expected improvement - used to find a promising point for a new evaluation. The `expected improvement' infill criterion takes into account the mean and variance of a predictive multivariate Gaussian distribution. The expected improvement algorithm has recently been generalized to multiobjective optimization. In order to measure the improvement of a Pareto front quantitatively the gain in dominated (hyper-)volume is used. The computation of the expected hypervolume improvement (EHVI) is a multidimensional integration of a step-wise defined non-linear function related to the Gaussian probability density function over an intersection of boxes. This paper provides a new algorithm for the exact computation of the expected improvement to more than two objective functions. For the bicriteria case it has a time complexity in $O(n^2)$ with $n$ denoting the number of points in the current best Pareto front approximation. It improves previously known algorithms with time complexity $O(n^3 \log n)$. For tricriteria optimization we devise an algorithm with time complexity of $O(n^3)$. Besides discussing the new time complexity bounds the speed of the new algorithm is also tested empirically on test data. It is shown that further improvements in speed can be achieved by reusing data structures built up in previous iterations. The resulting numerical algorithms can be readily used in existing implementations of hypervolume-based expected improvement algorithms. △ Less

Submitted 29 August, 2014; originally announced August 2014.

Comments: LIACS Technical Report

ACM Class: G.1.6; F.2.1

Convex Hull-Based Multi-objective Genetic Programming for Maximizing ROC Performance

Authors: Pu Wang, Michael Emmerich, Rui Li, Ke Tang, Thomas Baeck, Xin Yao

Abstract: ROC is usually used to analyze the performance of classifiers in data mining. ROC convex hull (ROCCH) is the least convex major-ant (LCM) of the empirical ROC curve, and covers potential optima for the given set of classifiers. Generally, ROC performance maximization could be considered to maximize the ROCCH, which also means to maximize the true positive rate (tpr) and minimize the false positive… ▽ More ROC is usually used to analyze the performance of classifiers in data mining. ROC convex hull (ROCCH) is the least convex major-ant (LCM) of the empirical ROC curve, and covers potential optima for the given set of classifiers. Generally, ROC performance maximization could be considered to maximize the ROCCH, which also means to maximize the true positive rate (tpr) and minimize the false positive rate (fpr) for each classifier in the ROC space. However, tpr and fpr are conflicting with each other in the ROCCH optimization process. Though ROCCH maximization problem seems like a multi-objective optimization problem (MOP), the special characters make it different from traditional MOP. In this work, we will discuss the difference between them and propose convex hull-based multi-objective genetic programming (CH-MOGP) to solve ROCCH maximization problems. Convex hull-based sort is an indicator based selection scheme that aims to maximize the area under convex hull, which serves as a unary indicator for the performance of a set of points. A selection procedure is described that can be efficiently implemented and follows similar design principles than classical hyper-volume based optimization algorithms. It is hypothesized that by using a tailored indicator-based selection scheme CH-MOGP gets more efficient for ROC convex hull approximation than algorithms which compute all Pareto optimal points. To test our hypothesis we compare the new CH-MOGP to MOGP with classical selection schemes, including NSGA-II, MOEA/D) and SMS-EMOA. Meanwhile, CH-MOGP is also compared with traditional machine learning algorithms such as C4.5, Naive Bayes and Prie. Experimental results based on 22 well-known UCI data sets show that CH-MOGP outperforms significantly traditional EMOAs. △ Less

Submitted 14 March, 2013; v1 submitted 13 March, 2013; originally announced March 2013.