Complexity and variety of modern multiobjective optimisation problems result in the emergence of numerous search techniques, from traditional mathematical programming to various randomised heuristics. A key issue raised consequently is how to evaluate and compare solution sets generated by these multiobjective search techniques. In this article, we provide a comprehensive review of solution set quality evaluation. Starting with an introduction of basic principles and concepts of set quality evaluation, this article summarises and categorises 100 state-of-the-art quality indicators, with the focus on what quality aspects these indicators reflect. This is accompanied in each category by detailed descriptions of several representative indicators and in-depth analyses of their strengths and weaknesses. Furthermore, issues regarding attributes that indicators possess and properties that indicators are desirable to have are discussed, in the hope of motivating researchers to look into these important issues when designing quality indicators and of encouraging practitioners to bear these issues in mind when selecting/using quality indicators. Finally, future trends and potential research directions in the area are suggested, together with some guidelines on these directions. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.  In the real world, it is not uncommon to face an optimisation problem with multiple objectives/criteria, namely, multiobjective optimisation problems (MOPs). These objectives are often conflicting, and there is no single optimal solution but instead a set of Pareto optimal solutions (termed a Pareto front in the objective space). A solution in the Pareto optimal set cannot be improved on an objective without degrading on some other objectives. For example, consider a car purchase problem where we want to buy a car with as good performance as possible but as low a price as possible. Apparently, we cannot find a single car that achieves the best on both objectives. Figure 1 gives all types of cars (solutions) available in the market. As can be seen, there exists a set of solutions (black points), each of which is not inferior to any solution on both objectives. We may only be interested in these Pareto optimal solutions as the remaining solutions (white points) are always outperformed by some of them. The growing interest in MOPs results in a variety of search techniques (called multiobjective optimisers hereinafter) [35, 48, 209] , ranging from exact to approximation methods [28, 145] , and from heuristics to metaheuristics [39, 72, 103, 108, 141] . These multiobjective optimisers aim to generate a solution set (a.k.a. a Pareto front approximation or an approximate set) that well represents the Pareto front of an MOP. Population-based non-numerical optimisers have been found promising in this regard, such as genetic algorithm [49], particle swarm optimisation [36] , and ant colony optimisation [2] . In such methods, each individual in the population aims to locate a unique trade-off among objectives, whereby they all together provide a representation of the whole Pareto front. A good Pareto front representation is of high importance in multiobjective optimisation. It presents a reliable and compact "picture" of the Pareto front and avoids overload in terms of the computation and the later decision-making process. From a representative solution set, the decision-maker can choose her/his favourable solution or uses this information to articulate preference that allows her/him to narrow down the search for a final choice. An important issue in multiobjective optimisation is to evaluate and compare the quality of solution sets, e.g., those generated by different multiobjective optimisers. However, in the presence of multiple (or even an infinite number of) optimal solutions, two solution sets may be incomparable. This contrasts with single-objective optimisation, where we can typically define the quality of a solution set by its best solution (i.e., the solution with the smallest or largest objective function value). In multiobjective optimisation, the quality of a solution set usually contains several aspects, e.g., the closeness to the Pareto front, the coverage over the Pareto front, and the uniformity amongst solutions. This substantially complicates the comparison between solution sets. A straightforward way to compare the quality of solution sets is visualisation. However, visual comparison fails to quantify the difference between solution sets and also becomes harder with more objectives involved. Quality indicators (QIs), 1 as a quantitative way of comparing sets, have arisen. They typically map a solution set to a real number that indicates one or several aspects of solution set quality, thus defining a total order amongst solution sets. QIs serve several purposes. They can be used to (1) compare competing multiobjective optimisers, reveal their strengths and weaknesses, and identify the most promising one; (2) monitor the search process of optimisers, optimise their parameters, and potentially improve their performance; (3) define the stop criterion of optimisers, especially those having a stochastic nature; and (4) explicitly guide the search by integrating into optimisers as selection criteria, e.g., instead of the Pareto dominance criterion. Over the last two decades, quality evaluation of multiobjective solution sets has gained much attention, not only in the fields of evolutionary computation, operational research, and
For a comprehensive review of quality indicators, in REF , some indicators are define and discussed using their quality facets as: convergence, spread, uniformity, and cardinality.
Quality Evaluation of Solution Sets in Multiobjective Optimisation: A Survey
CSUR
Computer Science