2.1 Classical IS Methods

Classical IS methods generally reduce the number of instances using the nearest neighbor rule. These approaches can be divided into condensation, edition, and hybrid methods. The condensed nearest neighbor (CNN) method retains points closer to the decision boundaries, while internal points are removed, since they do not affect the decision boundaries [¹⁴]. The edited nearest neighbor (ENN) approach preserves internal samples while removing points closer to the decision boundaries with class labels that are different from their neighboring points (i.e., noisy points) [²⁰]. The decremental reduction optimization procedure (DROP3) is a hybrid method that removes border points by first applying ENN to filter noisy instances, and then removes internal instances far from the decision boundaries [²¹]. Finally, the iterative case filtering algorithm (ICF) produces data clusters based on reachable and coverage sets, where points with a reachable set size greater than the coverage set size are removed [⁴].

2.2 Evolutionary IS Methods

An IS based on an EA is generally classed as a wrapper scheme. In this context, Kuncheva [¹⁵] proposed a GA with binary encoding, where the fitness function measured the error rate of the k-nearest neighbors (kNN) classifier.

In another work, Cano et al. [⁶] analyzed the performance of four binary-based representation EAs. The objective function adopted in this case was a weighted sum of the classification error and the reduction rate, with the same relative importance. Likewise, Garcia et al. [¹²] proposed a memetic algorithm that combined the heuristic approach of population-based algorithms with local search methods. The fitness function in this approach maximized both the accuracy and the reduction rate.

Aldana et al. [²] introduced a method based on an eclectic GA (EGA). This approach adopted a binary string encode in which two positive integers represented the number of randomly sampled instances. The EGAs objective function evaluated the reduction rate, and two constraints were considered: the error between the original and selected sets, and the proportion of elements between the quantiles of both sets.

Rosales-Pérez et al. [¹⁸] used a multiobjective EA to solve the IS problem. Their solution encoded the reduction technique and the hyperparameters of an SVM, and the two objective functions were the classification rate and the reduction rate.

4.1 Fitness Function

We propose a novel fitness function for use in an evolutionary IS algorithm. It has two components: (i) maximizing the similarity between the XO and XS PDFs; and (ii) minimizing the number of instances in XS.

This first component uses the Hellinger distance to measure the distributional divergence. The Hellinger distance between two densities p and q is defined as [⁸]:

and satisfies the property 0≤ℋ(p,q)≤1, where a value closer to zero indicates a higher similarity between the densities p and q. Hence, maximizing the similarity between p and q implies minimizing the Hellinger distance.

It is worth noting that p and q are univariate density functions. In contrast, the dataset XO usually contains instances in ℝd. To handle multivariate data, the Hellinger distance is calculated for each predictor variable for each class, to form the following matrix:

where ℋi,j≡ℋ(pi,j,qi,j) is the Hellinger distance between the original, pi,j, and approximated, qi,j, densities of the jth variable in the ith class. The PDF of a predictor variable is obtained using KDE, as described in Section 3. It is worth mentioning that the DPI rule is calculated on the jth dimension of the ith class of XO (for i=1,…,c and j=1,…,d ), and the resulting bandwidth hi,j is used to estimate the densities pi,j and qi,j.

The second objective is addressed by measuring the fraction of selected instances as:

where nS and nO denote the number of instances in XS and XO, respectively. This objective varies in the range [0,1], and a value close to zero indicates an adequate reduction rate.

Finally, the fitness function is used to combine the two criteria in (9) and (10) through a weighted sum, as follows:

where w∈(0,1) is a weight coefficient that expresses the importance of each objective. This weighted fitness function varies in the range [0,1], where an aptitude value close to zero indicates that XS achieves a high reduction rate and preserves the probability distribution of XO. We evaluated the impact of w by varying its value in the range [0.50,0.95] with steps of 0.05. Algorithm 1 presents a pseudocode for the evaluation of the fitness function.

Algorithm 1 Fitness function evaluation 

4.2 Evolutionary IS

In this work, we use a GA to minimize the objective function in (11). Each individual in the population is a fixed-length binary vector of size nO, where a value of 1 means that the corresponding instance in XO is selected, and 0 indicates the opposite.

Solutions are randomly initialized with a discrete uniform distribution in the range [0,1]. The parent selection strategy uses a two-way tournament approach. Next, a two-point crossover is performed in the recombination step to exchange the selected parents’ genetic information to generate new offspring. Then, based on the mutation probability factor, the mutation operator performs a bit flip in random positions of each offspring vector. An elitist strategy is also applied to ensure that the quality of the solution does not decrease over the generations. Finally, the GA returns the best individual in the last generation.

In this case, a population of 100 individuals was evolved over 2000 generations. The crossover and mutation probabilities were set to 0.9 and 1/nO, respectively.

5.1 Datasets

The datasets used in the experiments were obtained from the KEEL repository [¹] and the UCI Machine Learning Database [¹⁰]. Table 1 summarizes the characteristics of 40 small datasets (with no more than 5,456 instances). In order to test the performance of the proposed method with large datasets, two medium-sized datasets were considered: Magic Gamma Telescope (MGT) with n=19,020, d=10, and c=2, and Letter Recognition (LT) with n=20,000, d=16, and c=26.

5.2 Instance Selection Methods

The proposed approach, denoted as Fw (i.e., the proposed filter method with a specific weight value w), was compared against six other IS methods.

The first alternative approach was an evolutionary IS algorithm based on a wrapper scheme. This method used the same GA as the proposed approach but applied a fitness function that is commonly adopted in the literature, in which a weighted sum is used to combine the classification accuracy and the reduction rate with the same relative importance [¹⁷, ⁶, ¹²]. When evaluating the fitness function, the selected subset XS, given by a potential solution, is used to train the classification model, whereas the validation set is obtained as XV=XO−XS, and is used for measuring the classification accuracy. Two classifiers are considered, SVM and kNN, and the two variants of this approach are denoted as W_SVM and W_kNN (i.e., the wrapper method with SVM and kNN, respectively).

In addition, the soft margin parameter (C) and the bandwidth of the Gaussian kernel (γ) used for the W_SVM algorithm were found using the grid search method in the ranges C=[2−5,2−3,…,215] and γ=[2−15,2−13,…,23], with 5-fold cross-validation [⁷]. Appendix 7 lists the C and γ hyperparameters found.

The four remaining IS methods are CNN, ENN, DROP3, and ICF, as depicted in Section 2.1. The number of nearest-neighbors for W_kNN and the classical methods are set to k=3.

5.3 Performance Assessment

The performance of the IS methods was measured using four indices:

A t-times k-fold cross-validation method (where t=10 and k=5) was used to split the small datasets into training and test sets. This resampling process reduced the influence of randomness introduced by data splitting [²²]. To divide the medium-sized datasets, a 10-fold cross-validation technique was applied. The procedure illustrated in Fig. 1 was then performed on each fold.

Fig. 1 Evaluation framework for IS methods 

The reusability of the selected instances is related to the ability to train different classifiers without losing generalization. In this sense, in wrapper techniques, the subset XS could fulfill the classifier’s criteria to increase the accuracy, but loses similarity with the XO distribution when the number of instances is reduced; thus, XS may be useless for training other types of classifiers. To measure the reusability of XS, we use two types of accuracy:

Seven classifiers were considered when measuring the classification accuracy: classification and regression tree (CART), linear discriminant analysis (LDA), quadric discriminant analysis (QDA), na¨ıve Bayes classifier (NB), radial basis function network (RBFN), SVM, and kNN. Furthermore, max(3,n) hidden nodes were used for the RBFN architecture (where n is the number of the training instances), the number of nearest neighbors in kNN was set to k=3, and the hyperparameters of the SVM classifier were tuned for each subset using the grid search method depicted in Section 5.2 [¹¹, ³].

The non-parametric Kruskal-Wallis test, followed by a Bonferroni correction (α=0.05), was performed for multiple comparisons to determine the statistical significance between the proposed approach and the six IS methods in terms of the four indices listed above.

Additionally, the McNemar test (α=0.05) was used to statistically assess the accuracy of two classification models trained with XO and XS against the actual labels. It detects whether the difference between the misclassification rates is statistically significant. The null hypothesis establishes that the two predicted class labels, y^1 and y^2, have equal accuracy when predicting the actual class labels, y.

The testing platform used a computer with four cores at 3.5 GHz (Intel i7 4770k) and 32 GB of RAM. All the algorithms were developed in MATLAB 2018b [¹⁶], and the source codes are available upon request to the authors.

6.1 Instance Selection Performance on Small Datasets

Fig. 2 shows the average performance results on the 40 small datasets, for all of the classifiers specified in Section 5.3. For the proposed method, the Fw performance changed according to w, as expected. As w increases, the values of ACC and HDC also increase, while the values of E and RR decrease. However, the first five values of w (i.e., 0.50 to 0.70) produced the same efficiency (E=0.81), which was the highest for all of the IS methods. Furthermore, F65 and F65 yielded a value of ACC=0.70 and a fairly similar PDF preservation (HDC=0.93). F65 gave the best tradeoff, as it achieved a higher reduction rate (RR=0.85).

Fig. 2 Average performance results. They were calculated over the seven classifiers on the 40 datasets. Above each marker is shown the accuracy (ACC). The squared label shows the efficiency (E). The weight values (w) for the proposed approach Fw are shown at the bottom 

Moreover, the Fw variants gave better PDF preservation than the wrapper and classical IS algorithms. Notably, only ENN achieved the same Hellinger distance complement as F50 (HDC=0.90), the Fw variant with the lowest PDF preservation.

Of the classical IS methods, ENN obtained the best accuracy (ACC=0.76) and PDF preservation (HDC=0.90), although it had the worst efficiency of all of the methods (E=0.51) due to the low reduction rate (RR=0.24). CNN obtained the second-best value of accuracy (ACC=0.71) and PDF preservation (HDC=0.89) of the classical techniques, although it outperformed ENN in terms of efficiency (E=0.71) and reduction rate (RR=0.59). The hybrid methods yielded better reduction rates and efficiency but lower accuracies and PDF preservation than CNN and ENN. For instance, DROP3 had a better reduction rate (RR=0.83) and efficiency (E=0.74) than ICF, but the lowest accuracy (ACC=0.67) and PDF preservation (HDC=0.76) of all the methods. In contrast, ICF achieved a better accuracy (ACC=0.70) and PDF preservation (HDC=0.81), but a lower reduction rate (RR=0.72) and efficiency (E=0.73) than DROP3.

The wrapper methods achieved better efficiency and reduction rate than the classical techniques. Specifically, W_kNN obtained the highest efficiency (E=0.76), and W_SVM the best reduction rate (RR=0.87). However, in terms of the accuracy index, the wrapper methods did not outperform the classical ones. For example, W_kNN had the highest classification performance of the wrapper methods, but this was the same as for DROP3 (ACC=0.67), which was the worst of the classical algorithms in terms of accuracy. Regarding PDF preservation, the wrapper methods were again surpassed by CNN and ENN; however, W_kNN slightly outperformed ICF, while W_SVM had the second-worst HDC index, only outperforming DROP3.

Fig. 3 shows a detailed comparison of the results from F65 against all the IS methods. Tables 2 and 3 show the resulting p-value of the Kruskal-Wallis test for multiple comparisons of each performance index.

Fig. 3 Average performance results over the seven classifiers on the 40 datasets. Above each bar, the value of the corresponding measure. The dashed line marks the average accuracy obtained by the seven classifiers trained on the original datasets 

Regarding accuracy, F65 attained the third-best result, and was only outperformed by CNN and ENN; still, according to Table 2, there are no statistical differences from IS methods. Although F65 obtained the second-best value in terms of reduction rate, there was no significant difference from W_SVM, which achieved the highest performance for this index.

These methods also showed statistical differences from CNN, ENN, and ICF. Despite the high reduction rate of F65, it achieved the highest PDF preservation and showed statistical differences from W_kNN, W_SVM, DROP3, and ICF on the HDC index.

For the efficiency, F65 attained the highest value, and the results in Table 3 show that this method gave statistical differences from all of the IS methods except W_kNN.

These results suggest that for a specific value of w, the proposed approach selects a subset of instances that preserve the probability distribution of the original dataset with a high reduction rate. The selected subset is also useful for training distinct classification models with good generalization performance.

6.2 Instance Selection Performance on Medium-Size Datasets

The medium-sized datasets described in Section 5.1 were used to test the performance of the Fw method, with w=0.50 to give both objectives in the fitness function the same relative importance. The accuracy was calculated as the average over the results from the seven classifiers described in Section 5.3.

On the MGT dataset, the accuracy of the subsets was slightly better (ACC=0.81) than that obtained on the original dataset (ACC=0.80). The F50 method attained a regular efficiency (E=0.74) due to the poor reduction rate (RR=0.52) and a high PDF preservation (HDC=0.98).

Regarding the LT dataset, the accuracy on the original set (ACC=0.81) was slightly higher that attained on the subsets (ACC=0.79).

However, similarly to the MGT dataset, the efficiency was regular (E=0.73) due to the low reduction rate (RR=0.52) and the high PDF preservation (HDC=0.96).

6.3 Reusability of Selected Instances

The reusability results are shown in Fig. 4, where the Type 1 and 2 accuracies are displayed as pairs of box plots. The upper graphic relates to W_kNN, in which the Type 1 accuracy was measured only using the kNN classification, while the Type 2 accuracy was measured using the remaining six classifiers.

Fig. 4 Two types of accuracy results on the 40 datasets. Type 1: accuracy considering a single classifier, i.e., kNN (upper) and SVM (lower). Type 2: accuracy of six classifiers disregarding kNN (upper) and SVM (lower) 

Likewise, the lower graphic considers W_SVM, in which the Type 1 accuracy was measured only using SVM classification, while the Type 2 accuracy was measured using the remaining six classifiers.

For the proposed approach Fw, the accuracy improved with the values of the weights, i.e., the higher the weight, the better the accuracy. For each weight value, the median values for the two types of accuracy were quite similar. This suggests that the proposed method can give similar Type 1 and 2 accuracies, independently of the classifier, due to the criterion used to preserve the PDF of the original data.

The classical methods gave slightly better performance for Type 1 than Type 2, and ENN attained the highest accuracy due to its lower reduction rate. On the other hand, W_kNN and W_SVM obtained consistently better accuracy for Type 1 than Type 2. These results confirm that XS is biased towards the classifier characteristics in the wrapper method, limiting the reusability of the selected data for training other classifiers.

Finally, Fig. 5 shows the number of datasets for which there was no rejection of the null hypothesis in the McNemar test for each classifier. The higher the count, the more similar the classification accuracy between models trained with XO and XS.

Fig. 5 Counts of no rejections of the null hypothesis of the McNemar test 

In the proposed approach Fw, the higher the values of the weights, the lower the number of rejections. Thus, F95 obtained the highest number of selected subsets that did not show a statistical difference from XO for all of the supervised learning algorithms. Of the wrapper methods, W_kNN had a higher count of no rejections than W_SVM.

For the kNN classifier, W_kNN attained a significantly higher count than any other classifier, which was as expected since the selection criterion involved maximizing the accuracy of that specific classifier. W_SVM attained inferior results, even for the SVM classifier, where the count was not notably higher for the different classifiers, unlike the behavior of W_kNN for the kNN classifier.

Of the classical IS methods, ENN achieved a higher count in most cases, but gave a similar count to CNN for the kNN classifier, and was slightly outperformed by the same condensation method for RBFN. In terms of the number of counts for each classifier, the NB attained the highest value, and the QDA obtained the lowest counts for most IS methods.

These results reveal that the proposed method produces subsets of instances that can be reused to train different classifiers with a similar classification performance to that achieved from training with the original dataset.

6.4 Case Study on the Banana Dataset

Fig. 6 shows a case study carried out on the Banana dataset to compare the performance indices obtained by XO and XS. In terms of accuracy, F65 achieved a competitive result (ACC=0.74) for XO and obtained better performance than its counterparts except for ENN. For the reduction rate, F65 removed more than 85% of the samples (RR=0.86) and surpassed W_SVM, CNN, and ENN. It also yielded a competitive reduction rate with regard to W_kNN, DROP3, and ICF.

Fig. 6 Instance selection results on the Banana dataset. The average (ACC, RR, HDC, E) measures are shown in parenthesis inside each plot 

For PDF preservation, F65 gave the highest performance (HDC=0.98), and the data points in the feature space show that the selected subset follows the distribution shape of the original dataset, despite the high reduction rate. In contrast, the selected subset generated by ICF had holes and clumps, and produced the worst PDF preservation of all the methods compared here (HDC=0.79).

Finally, the F65 method gave the best efficiency (E=0.85) due to the high accuracy, reduction rate, and highest PDF preservation. ENN had the worst efficiency (E=0.44) as it gave the lowest value of the reduction rate (RR=0.12), despite its high accuracy (ACC=0.77) and PDF preservation (HDC=0.96).

Fig. 7 shows a comparison between the XO and XS PDFs for each class and dimension of the Banana dataset. The three IS methods yielded the highest HDC values in this case study, namely F65, W_SVM, and ENN.

Fig. 7 KDE results by class and predictor variable on the Banana dataset for XO and XS subsets obtained with F65, W_SVM, and ENN, respectively 

The results show that the proposed approach produced a selected subset that correctly matched the probability distribution of XO. In contrast, W_SVM and ENN gave a set of instances with slightly different distributions regarding XO, even when these methods obtained a lower reduction rate than F65.