1 Introduction

A definition proposed by Azzouz [¹] for a dynamic problem is: “dynamic multi-objective optimization problem (DMOP) is the problem of finding a vector of decision variables which satisfies a set of constraints and optimizes a vector of functions whose scalar values They represent objectives that change over time.” A DMOP can be defined as follows in Equation 1:

where x is the vector of decision variables, n is its size, Xn represents the solution space, f is the set of objectives to be maximized or minimized concerning time, g and h represent the set of inequalities and equalities, respectively, t represents the time or dynamic nature of the problem, and m is the number of objectives.

Ke Li [²] mentions that the recurring problems within single-objective and multi-objective optimization are determining a specific configuration for the control parameters and the selection of the correct genetic operators for each type of problem; this leads us to have different scenarios for solving problems, this becomes more complicated when the person is not an EA expert.

Search operators are of great importance in metaheuristics. Some operators are better for solving a specific problem. On the other hand, the selection and order of use of these operators can affect the performance of an algorithm [³].

In this case, we focus on selecting the correct genetic operator; adaptive operator selection, also called AOS, is used. The AOS is responsible for automatically determining which variation operator to use at a given time within a MOEA process.

Within the state of the art for AOS, the most recent works found correspond to methods based on multi-arm bandits applied to static multi objective optimization algorithms based on decomposition.

In this work, a new adaptive operator selection mechanism is proposed, which is based on the SARSA (λ) reinforcement learning technique. This new AOS mechanism has been incorporated into the DMOEA/D algorithm and compared against the state-of-the-art algorithm, which was also applied to the DMOEA/D algorithm.

2 Background and Related Work

The AOS comprises two main tasks [⁴]: credit assignment and operator selection. In the first task, the reward or weight of each operator is determined by their performance. The second task selects the best available operator.

Credits can be assigned in different ways depending on our method or algorithm. However, in general, it can be done based on the improvement of the children's fitness concerning the parents, and it can also be done based on a ranking of the operators.

Depending on how the operator selection task works, the AOS is divided into two groups: one encompasses all methods based on a probabilistic approach, and the other encompasses a multi-arm bandit approach (MAB).

In the multi-arm bandit approach, AOS uses the “multi-arm bandit (MAB) problem paradigm” [²], which considers each operator as an arm of a slot machine, each with an unknown reward probability. These methods seek to maximize the reward accumulated during the process and model these rewards to select the best operator (arm) at each moment.

In our previous work [⁵], we have applied “the Fitness-Rate-Rank-based Multi-Armed Bandit (FRRMAB)” [²], “Adaptive Operator Selection Based on Dynamic Thompson Sampling (DYTS)” [⁶], and “Adaptive operator selection with test-and-apply structure for decomposition-based multi-objective optimization (TAOS)” [⁷] methods to a dynamic version of the MOEA/D algorithm to observe its behavior.

Another of the most recent works in the area is “A novel bicriteria assisted adaptive operator selection (B-AOS) strategy for decomposition-based multi-objective evolutionary algorithms (MOEA/Ds)” proposed in 2021 by Wu Lin [⁸]. This approach employs two groups of operators; each group includes two genetic operators with different search patterns.

In addition, it uses two criteria, which emphasize convergence and diversity, to help select the appropriate operator. In the probabilistic approach, a probability is attached to each operator, and its selection process is similar to that of a roulette wheel; the operators with the highest probabilities will be those who will have a larger area on the roulette wheel and will be the ones who will have a greater chance of being selected.

The most popular methods within this group are Probability Matching [⁹] and Adaptive Pursuit [¹⁰]. Another work is “Adaptive crossover operator based multi-objective binary genetic algorithm for feature selection in classification” which uses a probability-based AOS within a multi-objective genetic algorithm to solve feature selection problems [¹¹].

The strategies mentioned above are applied to static multi-objective algorithms in the state-of-the-art. In this work, it has been decided to use the most recent strategy, “Adaptive operator selection with test-and-apply structure for decomposition-based multi-objective optimization (TAOS)” [⁷] and apply it to a dynamic multi-objective algorithm to compare the state-of-the-art strategy with the strategy proposed in this work.

2.1 Adaptive Operator Selection with Test-and-Apply Structure for Decomposition-based Multi-Objective Optimization (TAOS)

In this approach proposed by Lisha Dong in 2022 [⁶], the whole evolutive process is structured into several continuous sections, each designed to execute testing and application phases.

2.1.1 Test Phase

In the testing phase, each operator is tested in the same environment. The testing phase is divided into N parts, as shown in Figure 1: test₁, …, and test_N. All operators will be evaluated once in order. Therefore, the total number of function evaluations for the testing phase is defined as follows, as shown in Equation 2:

FEtest=K×N, (2)

where K denotes the number of operators, and the population size is defined by N. When an operator is applied in the search process, its impact needs to be measured; a successful update indicator (SUI) is introduced to assign credits to the operator. SUI is defined as follows as sampled in Equation 3:

SUI={1, if the generatedsolution is better thanat least one solutionin the population,0, and otherwise. (3)

Fig. 1 Test-and-apply structure [7] 

A child solution updates the first solution, and the SUI value changes from 0 to 1; if the child continues updating the solutions, the SUI will remain unchanged.

2.1.1 Apply Phase

When the testing phase has finished, the successful updates count (&W^) is obtained for each operator; this is shown in Equation 4:

SUCop=∑i=1SUIopi, (4)

where SUIopi indicates that the operator op is tested in evaluating the ith function of the testing phase. To select the operator, we compare them with each other.

The operator with the SUC with the highest value will be the one selected to be applied in the application phase of that segment. The number of function evaluations for the application phase in each segment is defined in Equation 5:

FEapply=FEtest×k, (5)

where k handles the resources for the testing and application phases, a shorter value of k, i.e., k<1, assigns more resources to the testing phase versus the application phase; a larger value of k makes fewer resources available for the testing phase.

2.2 Reinforcement Learning

Reinforcement learning is “learning how to do and map situations to actions to maximize a numerical reward signal. An Agent needs to be told what actions to take; instead, she must discover which actions yield the most significant reward by at-tempting them” [¹²].

In Figure 2, it can be seen that two important components of reinforcement learning are the Agent and the Environment. The Agent is the model that needs to be trained to make decisions.

Fig. 2 Interaction Agent-Environment 

2.2.1 SARSA (λ)

SARSA (λ) is a reinforcement learning algorithm used to train an Agent in an environment to make sequential decisions and learn to maximize a reward signal accumulated over time. It is a variant of the SARSA algorithm (State-Action-Reward-State-Action), which introduces an additional parameter called "lambda" (λ) to allow a trade-off between forward and backward learning.

The SARSA (λ) algorithm applies the TD (λ) prediction method to state-action pairs rather than just states, requiring a trace for each pair. Let Et(s,a) be the trace of the state-action pair s,a [⁹].

Next, algorithm 1 presents the general mechanism of SARSA (λ). The eligibility trace update is defined in Equation 6:

e(St,At)←λγe(St,At)+1. (6)

Algorithm 1. (λ) 

The calculation of the temporal error is defined in the Equation 7:

δ←Reward+γQ(St+1,At+1)−Q(St,At). (7)

The temporal error is used to update the Q values and improve the estimation of the quality of the actions in the different states. The update of the Q values is defined by the Equation 8:

Q(St,At)←Q(St,At)+αδe(St,At), (8)

where e(St,At) is the eligibility trace for the current state-action pair, δ is the temporal error, Reward is the calculated reward, α the learning rate, ε is the policy parameter ε − greedy, γ is a discount factor, and λ controls the amount of forward and backward learning.

3 Proposed Algorithm DMOEA/D-SL

This work proposes a new AOS method using a reinforcement learning technique called SARSA (λ) or SARSA Lambda. Furthermore, this new AOS has been integrated into a dynamic MOEA/D algorithm (DMOEA/D).

Two mechanisms proposed by Deb [¹³] have been added to the MOEA/D algorithm proposed by Zhang and Li in 2007 [¹⁴] to make an algorithm capable of working with dynamic problems, a change detection mechanism based on detectors, and the change response mechanism called “A”.

We have taken the multi-arm approach and used it as actions, in this case four variants of the differential evolution trader are being used as actions The SARSA Lambda mechanism has been incorporated into the main loop of DMOEA/D. The general structure of this integration is shown in algorithm 2.

Algorithm 2 . DMOEA/D-SL 

SARSA Lambda is initialized with the values corresponding to each of its parameters and is assigned to an object called an Agent; in this step, QTable is also initialized (line 3). For the episode loop, the main DMOEA/D loop is used (line 14), and for the step loop, the loop that runs through the population list (line 18) has been used.

In each turn of the main loop, St is initialized in the initial state S0 (line 15), and the Agent assigns At an action based on St. To select an action, it uses an ε-greedy policy, where a random number is generated between [0,1]; if the generated value is less than ε, an action (operator) is taken randomly, otherwise best action for the given state is selected (line 16).

We get all the values stored in the QTable for St and At and assign them to QVal_t (line 17). Once inside the loop that runs through the population, we apply the action (operator) obtained and produce a child called y′ (line 19). If y′ is better than any of the neighbors of individual i state S0 is assigned to St; otherwise, St will be assigned the following state (lines 26-52).

Subsequently, the following action, At+1, is calculated based on the state St+1, the value Qt+1 is obtained from the QTable for St+1 and At+1, the eligibility traces are updated with St and At, the reward for St and At is calculated, the delta value is calculated, and the QTable is updated. Finally, the state St and the current action At are updated.

This process is repeated until the stopping criterion has been met. It should also be considered that in each problem change, the QTable must be reinitialized (line 67).

4 Computational Experiments

Table 1 shows the eight dynamic multi-objective benchmark problems of 2 and 3 objectives used in this experiment. For each algorithm and front of the dynamic multi-objective problem, 30 independent runs were conducted.

The objective of the experimentation is to compare the proposed algorithm against the state-of-the-art algorithm called “Adaptive operator selection with test-and-apply structure for decomposition-based multi-objective optimization (TAOS)” [⁶]. Table 2 shows the parameters used for each algorithm used. The algorithms were implemented in the Java language.

The values of the parameters for MOEA/D have been taken from state-of-the-art, and the values of the parameters of the SARSA Lambda Agent have been obtained by assigning values arbitrarily by performing multiple experiments to determine the current values.

5 Conclusions and Future Work

This work proposes a new adaptive operator selection strategy using a reinforcement learning agent. This SARSA Lambda reinforcement learning strategy has been integrated into a dynamic multi-objective decomposition-based algorithm called DMOEA/D-SL. Furthermore, a state-of-the-art adaptive operator selection strategy, in this case, “Adaptive operator selection with test-and-apply structure for decomposition-based multi-objective optimization (TAOS),” has also been integrated into a dynamic multi-objective decomposition-based algorithm.

In the case of the proposed algorithm, the Agent learns to select the best genetic operator at a given moment, even when the definition of the problem changes over time. Extensive experimentation has been performed, and the results have been evaluated with three metrics: hypervolume, generalized spread, and inverted generation distance.

The Wilcoxon test has been applied with a significance level of 5%. The Wilcoxon test suggests that experimentation results are favorable in two metrics for DMOEA/D-SL. These results suggest that the algorithm produces high-quality solutions with a good approximation to the Pareto front and good dispersion.

Future work proposes exploring the parameters of the SARSA Lambda Agent more broadly to achieve better results in the three metrics used. On the other hand, using more than four genetic operators would test the behavior of the strategies used in this work and their performance.

Finally, other reinforcement learning strategies can also be considered to improve the quality of the solutions generated by the algorithm further.