# IEEE WCCI (CEC) 2024 Competition on Multi-Objective Black-Box Optimization Benchmarks in Human-Powered Aircraft Design The Python source code and related paper are available at the following URL: code: [https://github.com/Nobuo-Namura/hpa](https://github.com/Nobuo-Namura/hpa) paper: [Nobuo Namura. Single and Multi-Objective Optimization Benchmark Problems Focusing on Human-Powered Aircraft Design. arXiv:2312.08953, 2023.](https://arxiv.org/abs/2312.08953) ## Announcements - Jul. 12, 2024: Competition results have been posted, see [Results](#results) - Jun. 16, 2024: Schedule for the result announcement has been posted, see [Competition Schedule](#competition-schedule) - May 28, 2024: Update of submission details, see [Submission](#submission) - Apr. 21, 2024: Two rules have been added, see [Remarks](#remarks) (**==rules added are highlighted==**) - Apr. 7, 2024: Award information has been disclosed, see [Award](#award) - Mar. 13, 2024: Submission via Google Forms or e-mail has been opened, see [Submission](#submission) - Jan. 21, 2024: Change in data submission method, see [How to Participate](#how-to-participate) ## Results [Here](https://drive.google.com/file/d/1bqrmnp8ygaavZPDFM9L1daH4i0CLycoU/view?usp=sharing) is the material containing the problem explanation and the summary of results used in the result announcement session. |Ranking|Score|Algorithm|Team|Material| |:----|:----|:----|:----|:----| |1|0.02606 |IBEA with gene level hybrid crossover |Keigo Momma (IBM Japan), Naruhiko Nimura, Shu Ota, Kento Kaneko, Takeshi Sawada, Ryutaro Onishi (The University of Tokyo), Akira Oyama (Institute of space and aeronautical science)| [slides](https://drive.google.com/file/d/1G2oBLadWpCKx7UK1Iub1VJ4a8HVAhH_W/view?usp=sharing) | |2|0.03129 |Pareto Front Modeling convex DEA Selection GA|Mamoru Doi (Mitsubishi Electric Corporation)| unavailable | |3|0.03234 |SMS-EMOAII|Yuki Azuma, Takato Kinoshita, Naoki Masuyama, Yusuke Nojima (Osaka Metropolitan University)| [slides](https://drive.google.com/file/d/1E9vmgkUFniEkPgPewSBTrzVn6xbpUksz/view?usp=sharing) | |4|0.03270 |Extension of NSGA-II|Rin Ito, Hideyuki Masui (Mitsubishi Electric Corporation)| [slides](https://drive.google.com/file/d/1nGa8V_Z49R6beSORNwT1848zXrOPljxW/view?usp=sharing) | |5|0.03712 |2 stage optimization |Kuniaki Satori (Mitsubishi Electric Corporation)| [slides](https://drive.google.com/file/d/1x2HjAe0GMQoWeFgfSzwdqn4H9jdinX4o/view?usp=sharing) | |6|0.04041 |NSGA-2-GSO|Koki Yamada, Takeshi Uchitane (Aichi Institute of Technology), Kazunori Iwata (Aichi University), Nobuhiro Ito (Aichi Institute of Technology)| [slides](https://drive.google.com/file/d/1DF7PwJpTx1MHsEVdxiVeZbkv_pj_z8XE/view?usp=sharing) | |7|0.05335 |SFA/DE-NSGA-II|Kei Nishihara, Yuma Horaguchi, Takashi Ikeguchi, Masaya Nakata (Yokohama National University)| [slides](https://drive.google.com/file/d/1j3owW7G8IgMchBSj3o97L2UdCAHu52xz/view?usp=sharing), [SFA/DE](https://www.sciencedirect.com/science/article/abs/pii/S221065022400049X) | |8|0.39054 |Wavelet-like grid search|Anonymous team| [slides](https://drive.google.com/file/d/1IShlpj0k0vHBq1SZuFpUdh_S9LXYBTcw/view?usp=sharing) | ## Competition Schedule The schedule for this competition is as follows: - Dec. 15, 2023: Problem Announcement - May 31, 2024 **==(Anywhere on Earth)==**: Result submission deadline - Jul. 1, 2024: Competition result announcement during the IEEE WCCI 2024 ### Result announcement session The competition results will be announced on July 1, 2024, from 16:40 to 18:00, in Room 213 at the Pacifico Yokohama during IEEE WCCI 2024. Participants of the competition are encouraged to attend this result announcement session. Each team's representative should give a two-minute presentation of their algorithm using the submitted materials. Presentation materials will be projected from the organizer's PC, so there is no need to bring your own computer. ### Schedule on July 1, 2024 | Room 213 | Contents | Presenters | |-------------|--------------------------------------------------------|--------------------------------| | 16:40-16:45 | Opening | Nobuo Namura (Fujitsu Limited) | | 16:45-17:00 | Introduction to HPA problems | Nobuo Namura (Fujitsu Limited) | | 17:00-17:20 | Algorithm presentation (2 minutes for each team) | Participants | | 17:20-17:45 | Result announcement & summary of results | Nobuo Namura (Fujitsu Limited) | | 17:45-18:00 | Discussion, Q&A | - | ## Introduction This is an optimization competition aimed at fostering collaboration between industry and academia and promoting research, organized by the Evolutionary Computation Society of Japan. The Society holds annual competitions using real-world problems faced in industrial settings. Although this is the first time the competition is being held at an international conference, the goal is to encourage the emergence of more approaches and to stimulate more active collaboration between industry and academia. In this competition, we will present a problem of human-powered aircraft engineering design optimization, which is more practical and less restrictive than traditional benchmark problems. This page provides an explanation of the background and details of the problem, followed by guidance for participating in the competition. ## Background Evolutionary algorithms are extensively applied in multi-objective black-box optimization across various domains, including engineering design, facility operation, material development, drug discovery, and machine learning. Even though benchmark problems have a significant influence on the research direction of algorithms, many artificial benchmark problems possess unique characteristics that differ from real-world problems, leading to concerns about overfitting algorithms to these benchmarks [3]. In recent years, practical benchmark problems [4],[5] based on real-world problems have been used as alternatives to these artificial benchmark problems. However, such benchmark problems are often limited to relatively low dimensions or require computationally expensive numerical simulations [6]-[8], which can limit the number of function evaluations. To overcome these limitations, we introduce practical benchmark problems [1] focused on the engineering design optimization of human-powered aircraft. These benchmarks exhibit five key features: 1. Real-world engineering problem resemblance 2. Variety in Pareto front shapes 3. Scalable design variable dimension 4. Transformation of practical constrained problems into unconstrained formats 5. Low computational cost ## Human-Powered Aircraft (HPA) HPA gain propulsion by the pilot pedaling like a bicycle to rotate a propeller. HPA typically have a wingspan of 20-35 [m], comparable to that of a passenger aircraft, yet their mass is only about 25-35 [kg]. They often feature wings made of foam and balsa wood, covered with heat-shrinkable film, and supported by carbon fiber reinforced plastic (CFRP) pipe frameworks. The competition focuses on optimizing the main wing design of an HPA, including wing shape, lamination of CFRP pipe, dihedral angle at the wing root, wire tension, and payload. The wing segmentation number $n$ alters the problem's dimension, set at $n=4$ for this competition, as illustrated in Fig. 1.

Fig. 1. Human-powered aircraft in flight (left) and wing segmentation (right)

## Benchmark Problem Definition The HPA problems [1] in this competition are formulated as multi-objective minimization problems with box constraints: $$ \begin{array}{lr} \text{minimize} & \quad & {\mathbf F({\mathbf x}_l)} = (f_1({\mathbf x}_l), \ldots, f_M({\mathbf x}_l)) \\ \text{subject to} & \quad & {\mathbf x}_l \in [0, 1]^{D_l} \\ \end{array} $$ where, $l \in \{0,1,2\}$ represents the problem's difficulty level, with higher values indicating greater design variable freedom. We classify problems as HPA $MNL$ - $l$ , where $M$ is the number of objectives, $N$ is the number of constraints (excluding box constraints), $L$ is the problem index, and $l$ is the difficulty level. The original paper proposed 60 problems (20 types $\times$ 3 levels) with 1-9 objectives. In this competition, we chose 10 problems with two and three objective functions at $l=1$. Table 1 summarizes names of the selected problems and the design variables dimension $D_1$ Further details are in reference [1]. Table. 1 Problems and dimensions in this competition | Problem | $D_1$ | |-----------|-------| | HPA201-1 | 33 | | HPA202-1 | 31 | | HPA203-1 | 34 | | HPA204-1 | 33 | | HPA205-1 | 33 | | HPA301-1 | 33 | | HPA302-1 | 31 | | HPA303-1 | 34 | | HPA304-1 | 33 | | HPA305-1 | 33 | ## Competition Rules ### General settings * Maximum function evaluations: 20,000 * Independent runs: 11 per problem ### Performance metric The Inverted Generational Distance Plus (IGD+) metric [9] is used for algorithm performance comparison. Preliminary reference points for IGD+ are provided by [2], derived from six algorithms under hot-start conditions in [1]. Final performance will be evaluated using non-dominated solutions (NDSs) among solutions from all participants and the preliminary reference points, as modified reference points. Note that not all NDSs will be selected as reference points. We will select the NDSs with the maximum minimum Euclidean distance from the selected reference points in the normalized objective space until we reach $200M$ points or the number of preliminary reference points. The calculation of IGD+ is conducted after the objective space has been normalized, using the utopia and nadir points in the reference points. ### Ranking criteria The mean IGD+ across 11 runs for each problem will be computed and then their geometric mean across all problems will be used as a score for ranking. (lower scores indicate better performance). Let $s_{i,j}$ be IGD+ in the $i$ -th run for the $j$ -th problem, the score for ranking is calculated as: $$ S = \Biggl( \prod_{j=1}^{10} \frac{1}{11} \sum_{i=1}^{11} s_{i,j} \Biggr)^{\frac{1}{10}} $$ ### Remarks * Treat the problem as a black box; the only information you can use is the number of objectives and variables. * Apply the same algorithm and parameters to all problems (parameters such as population size and mutation probability can be functions of the number of objectives and variables). * Begin each run with a random state, using different random seeds (design of experiment without prior knowledge of the problem is permissible). * Avoid sharing solutions between different problems. * Refrain from using information from the reference points of IGD+ (e.g., normalizing the objective space using these points is not allowed in algorithms; only solutions obtained in each run may be used for normalization). * Participants may use either newly proposed algorithms or their previously published algorithms. * ==**The same participants can join multiple teams, but each team must have a different head who will be registered as the first member on the application form.**== * ==**If no one from the team registers to participate in [IEEE WCCI 2024](https://2024.ieeewcci.org/), they will be given a reference rank and will not be eligible for awards.**== ## How to Participate ### Function evaluation 1. Download the Python source code for HPA problems from [GitHub](https://github.com/Nobuo-Namura/hpa) 2. Run the code using your algorithm for the competition problems listed in Table 1 - Use the parameters "n_div=4, level=1, NORMALIZED=True" for all problems - All solution data must be included in the data submission; refer to [Data format](#data-format) beforehand - MATLAB users might need to use 'py.importlib.import_module' for evaluation - Each function evaluation may require about 0.13 seconds, and thus a total of 80 CPU hours are needed Python example: ``` from hpa.problem import HPA201 import numpy as np func = HPA201(n_div=4, level=1, NORMALIZED=True) x = np.random.uniform(size=func.nx) f = func(x) ``` ### Submission Please submit the following data via [Google Forms](https://docs.google.com/forms/d/e/1FAIpQLScLwgU6Zi-LDPaWL35lfE642ihiZ7Up9YF7TJzXLjE457Xbfg/viewform?usp=sharing) or e-mail ([sig-rbp@googlegroups.com]()). 1. Names, affiliations, e-mail addresses, and registration for WCCI (completed/planned/absent) of the participants 2. All solution data in the specified format (zip of CSV files); since the file size is about 1GB, please send us a link to the zip file if you submit it via email 3. Name of your algorithm; this will be used when posting the ranking and scores on this website after the results are announced 3. Two-page presentation slides (PDF) describing your algorithm; participants will be asked to make short presentation (within 2 minutes) during a session at WCCI 4. Reference information about any papers if your algorithm has been or will be published (optional) 5. Respond to the consent confirmation below in the specified format (delete either "yes" or "no" as appropriate) * May we publish the names and affiliations of the participants on this website? (yes/no) * May we publish the two-page presentation slides on this website? (yes/no) * May we publish the reference information on this website? (yes/no) ### Data format All solution data, which include the values of design variables and objective functions for each run, should be submitted in CSV files. The file name should consist of the problem name (e.g., HPA201-1) and the run number (e.g., 5), formatted as 'HPA201-1_5.csv'. Each file should be formatted as follows | evaluation | generation | nondominated| f1 | f2 | x1 | $\cdots$ | x33 | |------------|------------|-------------|----|----|----|----------|-----| | 1 | 1 | 0 | | | | | | | 2 | 1 | 0 | | | | | | | $\vdots$ | $\vdots$ | $\vdots$ | | | | | | | 20000 | 200 | 1 | | | | | | - 'evaluation': the number of function evaluation at which the solution is obtained. - 'generation': the generation number at which the solution is obtained. If your algorithm does not use this, enter the same value as 'evaluation'. - 'nondominated': a flag to identify non-dominated solutions in each run, where 1 indicates a non-dominated solution and 0 indicates a dominated one - f1, f2, (and f3, if applicable): values of the objective function - x1, $\cdots$, x33: values of the design variables (the number of variables varies depending on the problem) ## Award The top three teams in the ranking score will be awarded the following prizes from IEEE CIS: - First place: $500USD - Second place: $300USD - Third place: $200USD ## Contact If you have any questions or uncertainties regarding the competition, please contact the organizers of the Evolutionary Computation Competition. Email: [sig-rbp@googlegroups.com]() ## Organizers * Nobuo Namura, Fujitsu Limited, Japan * Ryuji Mishima, Tokyo Institute of Technology, Japan * Takato Kinoshita, Osaka Metropolitan University, Japan * Yuki Tanigaki, National Institute of Advanced Industrial Science and Technology, Japan * Naoki Hamada, KLab Inc., Japan * Yusuke Nojima, Osaka Metropolitan University, Japan ## References 1. [N. Namura., "Single and multi-objective optimization benchmark problems focusing on human-powered aircraft design." arXiv:2312.08953, 2023.](https://arxiv.org/abs/2312.08953) 2. Source code ([https://github.com/Nobuo-Namura/hpa](https://github.com/Nobuo-Namura/hpa)). 3. H. Ishibuchi, Y. Setoguchi, H. Masuda, and Y. Nojima, "Performance of decomposition-based many-objective algorithms strongly depends on Pareto front shapes," *IEEE Trans. Evol. Comput.*, vol. 21, no. 2, pp. 169-190, 2017. 4. R. Tanabe and H. Ishibuchi, "An easy-to-use real-world multi-objective optimization problem suite," *Appl. Soft Comput.*, vol. 89, pp. 106078, 2020. 5. Z. Wang, C. Gehring, P. Kohli, and S. Jegelka, "Batched large-scale Bayesian optimization in high-dimensional spaces," in *Proc. Int. Conf. Artif. Intell. Stat.*, pp. 745-754, 2018. 6. S.J. Daniels, A.AM Rahat, R.M. Everson, G.R. Tabor, and J.E. Fieldsend, A suite of computationally expensive shape optimisation problems using computational fluid dynamics, in *Proc. Int. Conf. Parallel Probl. Solving Nat.*, pp. 745-754, 2018. 7. V. Volz, B. Naujoks, P. Kerschke, and T. Tusar, "Single-and multi-objective game-benchmark for evolutionary algorithms," in *Proc. Genetic Evol. Comput. Conf.*, pp. 647-655, 2019. 8. C. He, Y. Tian, H. Wang, and Y. Jin, "A repository of real-world datasets for data-driven evolutionary multiobjective optimization," *Complex Intell. Syst.*, vol. 6, pp. 189-197, 2020. 9. H. Ishibuchi, H. Masuda, Y. Tanigaki, and Y. Nojima. "Modified distance calculation in generational distance and inverted generational distance," in *Proc. Int. Conf. Evol. Multi-Criterion Optim.*, pp. 110窶・25. 2015.