nueva página del texto (beta)
Inglés (pdf)
Artículo en XML
Referencias del artículo
Traducción automática
Enviar artículo por email
Citado por SciELO 
Similares en
SciELO 
Permalink
1 Introduction
Day by day, humans are taking advantage of knowledge that we acquire to solve our tasks more easily, one of them is known in pattern recognition as Classification [10]. With great precision and according to desired parameters we can classify experiences, people, tangible or intangible things thanks to the knowledge we have previously acquired, but what happens when suddenly the nature of the data has changed?
This phenomenon is named concept-drift, one of the main challenges of working with data streams [13]. The detection of categories or classes has had an impact on various sectors, both in research and in industry; as an example the companies that manage their sales on Internet.
This has been acquiring greater importance since the data that is generated every day is more and larger, so it is necessary to generate efficient tools for data analysis and decision making process over the data streams [6].
The nature of data streams is dynamic and therefore they are always evolving [2]. Given this situation it is not so easy to treat them with simple and static strategies. The main challenges we face when working with data streams are:
Infinite length: the analysed data is potentially infinite so it would be impossible to store all this information.
Concept-evolution: is the phenomenon that occurs due to the presence of new unknown classes in the data.
Concept-drift: are changes in input data attribute values that indicate a change in data stream behaviour.
Feature evolution: occurs when new features appear in the data stream and initial features may disappear.
Data Mining of dynamic data stream is of great importance in the field of Machine Learning. These scenarios require adaptive algorithms that are able to process the input instances in real time, adapt to potential changes in the data, use limited computational resources, and be robust to atypical events that may occur [16]. A great deal of Machine Learning approaches have been developed to target concept-drift detection, such as neural networks [28, 18], active learning [12], instance based classifiers [5], Bayesian techniques [15], and ensembles classifiers [12], among others.
Data stream clustering approach is being widely applied in data stream scenarios. Clustering naturally adapts to speed and memory restrictions of data stream learning since it can maintain the information summarized of the cluster without the need of storing all the instances observed in a data stream.
In current literature a great deal of work addressing clustering in this type of scenarios can be found [14]. The works mentioned above provide the main ideas for the proposed solution in this document, which aims to implement a model for data stream classification that can handle concept-drift.
The rest of the paper is organized as follows: in section 2 materials and methods are described. In the following section, 3, the proposed solution and its flow chart are introduced. In section 4, the results and discussion are presented and finally in Section 5, the conclusions and proposals for future work are provided.
2 Material and Methods
This section is composed of two subsections necessary for the development of the proposed solution. In Section 2.1 the materials used in this paper are described. These materials consist of three data sets commonly used for data stream classification and the framework used to run some experiments.
In Section 2.2, the NACOD algorithm is described, which is the classifier used in this proposal. In addition, concepts such as: k-means, Silhouette coefficient and Global Hybrid Online Similarity are described.
2.1 Materials
2.1.1 Algorithms and Data Sets
This section describes the algorithms and data sets used during the experimental phase and explains the main reasons why these were selected.
MOA: to evaluate the results obtained, a comparison with other data stream classification algorithms was carried out. The MOA implementation of these algorithms was used. MOA [3] is an open source framework for massive online analysis. This framework provides several tools to deal with data stream scenarios, such as the data stream version of Naïve Bayes [3] and Hoeffding Tree [9]. The proposed methodology has been tested with three real data sets, that are often used in current works of data stream classification mainly because their size is suitable to simulate data stream scenarios. The selected data sets are:
-
– KDD Cup 1999: this is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, includes a wide variety of simulated intrusions in a military network environment. The size of the original data set has more than four million instances but commonly only
– Forest Cover Type: describes seven forest cover types (classes) on a
-
– Statlog (Shuttle): this data set has
The number of classes in the selected data sets varies between
2.2 Methods
The methods used in this article include clustering, specifically k-means, Silhouette coefficient, the associative classifier: NACOD, the similarity operator: GHOS.
2.2.1 Clustering
Clustering is an unsupervised learning task; it means that it works with data without labels. It consists of distributing a set of instances into groups according to their similarities.
The major difference with respect to classification is that the number of groups is unknown before starting the learning process and the task is to create them [24]. In this work, the k-means algorithm will used. Next sub-section introduces theory of this algorithm.
2.2.2 K-Means
Proposed by MacQueen in
The algorithm does not conclude until there is no need to move any observation point to a different cluster [21]. Due to its simplicity, k-means [20] is one of the most used algorithms in clustering. It is known for its faster computational speed and superior performance on large data sets in comparison to other clustering techniques [29].
2.2.3 Silhouette Width Criterion
Silhouette width criterion was first defined in [25], also called Silhouette coefficient. It is based on geometrical considerations about separation and compactness of clusters and it helps to validate the micro-clusters and decide if they must be added to the model.
Consider the
Figure 1 represents an example of how Silhouette coefficient of an
2.2.4 NACOD
Recently, Villuendas-Rey et al. [26] introduced the Naïve Associative Classifier for Online Data (NACOD), a decision-making algorithm within the associative approach. This classifier can deal with some data complexity issues such as: missing values, hybrid and incomplete data, which significantly complicate the operation of the learning algorithms. NACOD is divided in three phases: training, classification and updating.
Training phase. It consists on storing relevant knowledge information about an initial set of labeled data.
Suppose the following data sets as shown in Figure 2. The main objective in this section is to create an archive to store knowledge about known classes and attributes. Figure 2 shows how this information is handled. This archive also stores a prototype of each of the classes seen so far, like that: the class prototype is selected by using class mean for numeric attributes and class mode for categorical attributes.
For each numeric attribute, the classifier makes the sum of its complete values, and the squared sum of differences between current values and current mean. The current mean is computed by dividing the sum of complete values by the count of complete instances. The squared sum of differences allows the classifier to compute an estimation of the standard deviation of the attribute values, without the need of storing past data.
Figure 3 represents the archive corresponding to the training set described in Figure 2. NACOD uses the prototypes in the archive to compute similarities among instances. Instead of comparing the instance to classify with respect to the entire training set, it compares it with respect to the class prototypes.
Classification phase. To classify a new instance and make a decision, NACOD first computes its overall Global Hybrid Online Similarity (GHOS) with respect to the classes in the current training set (prototypes), equation 2. Then, it will assign the instance to the class with maximum similarity.
Let
where:
The meaning of
For categorical attributes,
Update phase. NACOD will be classifying as new instances arrive, and then the classified instances will be used for updating the model.
Two possibilities exists: the first is that the instance belongs to a known class, and the other is that the instance does not belong to any of the existing classes. In the first situation, NACOD increases the number of instances seen so far of the corresponding class, as well as updates the maximum, minimum, sum of values and sum of squared differences for each numeric attribute, and the current values and occurrences for categorical attributes in the prototype of the corresponding class.
And in the second case, the instance will be stored by NACOD as a new class prototype; the sum of values and sum of squared differences for the new class will be initialized, as well as the values and appearances of the categorical values. NACOD pseudocode is presented in Algoritmic 1.
2.3 Concept-Drift
Given the non-stationary nature of data streams, the concept represented by the data that is being observed can change over time.
When this changes affect the relation between the input variables and the target variable, we are in the presence of concept-drift [11, 19, 12]. Formally, concept-drift between time point
where
3 Proposed Solution
The proposal is divided into an offline phase and an online phase. During the offline phase (training) a clustering process is performed, therefore, Subsection 2.2.1 introduces the clustering algorithm used. In Subsection 2.2.3, the Silhouette coefficient was presented, which is used to determine the cohesiveness of a cluster.
The online phase uses the NACOD associative classifier to classify the arriving instances; the theory behind this classifier was presented in Section 2.2.4. Examples not described by the current model are marked as outliers and stored in a buffer to be used later to address one of the main challenges in data stream classification: concept-drift, explained in section 2.3, or perhaps discard them if they are outliers.
3.1 Problem Formalization
We first formalize the problem of our data sets for data stream classification here. A data stream
As each instance arrives, we can collect
The training data set
3.1.1 Offline Phase
In this first phase, performed offline, that can be considered the training phase of the algorithm, a set of labeled instances will be needed to create the initial model.
As suggested in [1] an
During the update step, which will further explained in Subsection 3.2.4, these initial micro-clusters will be adjusted to reflect the changes in the data stream. Each micro-cluster will be denoted by
Also the micro-clusters need to have an indicator of the timestamp when it was last updated. To this end, the index of the last aggregated instance will be used as the micro-cluster timestamp:
Each micro-cluster is described by
This representation is an extension of the one used in [8]. Using these measures it is possible to calculate other important metrics required for the implementation of this proposal, such as the standard deviation of each feature, the distance between micro-clusters, and the standard deviation of the distances between a centroid and its instances. To calculate the centroid of a micro-cluster the following equation was used, as proposed in [22]:
The equations used to calculated each of the other elements that describe a micro-cluster are the following:
Other statistics must be calculated to use during the classification step. The NACOD classifier, uses standard deviation to calculate similarity of numeric features. Assuming the possible infinite length of a data stream, it is impossible to store all the instances that have been seen; instead summarise of the data are stored. Therefore, an estimation of the standard deviation will be calculated using the squared sum of differences. For this purpose, the squared sum of differences
3.2 Online Phase
Two possibilities exist when an instance arrives:
The instance is classified (correctly or incorrectly).
The instance is sent to a temporary buffer, because is not explained by the current model.
3.2.1 Classification
During this phase, data will be read in chunks of 2, 000 instances and the label of the new incoming instances of the data stream will be predicted. In this proposal, the NACOD algorithm [26] will be used. Unlike the original NACOD proposal [27], which only handles one standard deviation per class, our proposal uses several standard deviations per class, i.e. one for each micro-cluster that describes the class.
This, allows a better description of the distribution of the instances in the feature space. A classification threshold based on the GHOS operator was used to decide if a new instances should go through the classification process or should be sent to the temporary buffer. Several values were tested for this threshold, but the best results were obtained with values between
The Global Hybrid Online Similarity (GHOS) is computed between the new instance and the centroids of all current micro-clusters. If the obtained value of GHOS is greater than or equal to the classification threshold proposed, the instance is classified and assigned to the class of the micro-cluster whose centroid has the maximum similarity with the instance and its information will be used to update the model. Otherwise, the instance will be store in a temporary buffer for further analysis. The general idea of the classification phase is shown in Figure 6.
3.2.2 Buffer Analysis
The instances that were not explained by the current model, i.e. instances stored in the temporary buffer, will be studied to determine if they are outliers or hints of the beginning of a concept-drift. In the second case, those instances will be used to model new micro-clusters that allow the model to incorporate this new knowledge. The process on concept-drift detection will be executed at the final of each data chunk of 2, 000 instances. Figure 7 shows the novel class detection process. Two possibilities exist when temporary buffer is analysed:
Single or multiple instances can be outliers.
A representative number of instances can be declared as a class extension, known as a concept-drift.
To determine which one of the previous cases is, is necessary to have cohesive sets of representative instances. To identify these sets, a cluster process will be applied to the temporary buffer when a considerable number of instances stored in it is reached.
This number
where
If the new micro-cluster is valid, i.e. representative and cohesive, the distance between the centroid of the new micro-cluster
That means the decision boundary has changed, this explains why NACOD did not classify well. Otherwise, the temporary buffer is not modified and the instances are kept for further analysis. This may mean a concept-drift or outliers. To prevent a memory overflow, the user must define a limit number of observations to be stored in the temporary buffer.
3.2.3 Buffer Update
The temporary buffer needs to be updated: instances that have been too long in this buffer are removed. For this, the timestamp of the instances is also stored.
At the end of each data chunk, the temporary buffer is checked and the instances whose difference between their timestamp and the current timestamp is greater than the defined data chunk size are considered outdated instances and will be eliminated.
3.2.4 Model Update
When new instances
Linear
Notice that before calculating equation 15,
Then a process of clustering using the k-means algorithm has to be executed and the Silhouette coefficient is calculated for each resulting cluster in order to assess if there are valid clusters to be added to the model. If a micro-cluster is valid, it has to be added to the model as an extension of a class. If the micro-cluster is not valid, it is discarded but the instances are kept in the temporary buffer.
4 Experimental Results
This Section explains what evaluation approach and measure metric were used. Also shows the results obtained from the experiments performed with the proposed solution presented in Section 3 and a discussion of them. Accuracy vs data chunks graphs to observe the performance of the algorithm as time goes by are provided.
Exist several approaches that are commonly used to estimate the performance of models when the data are not stationary. The objective of these approaches is to leave the last part of the data for testing and preserve the temporal order of observations although they do not make full use of all data.
In this article, we use Prequential approach, or Interleaved Test-Then-Train [7], because is one of the most common evaluation approaches in incremental data streams [11], in which first an observation (or a set of observations) is used to test and after, this observation is used to update and train the model. Figure 8 depicts Prequential approach, the data are split into
As classification of data stream is not a static task, researchers evaluate over every data chuck how the classifier is working [4]. The measure implemented in this research is accuracy, is one of the easiest and simplest performance measure. It is determined as the ratio of correct predictions to the total number of test patterns [23].
As mentioned before, Table 3 refers to the parameters used in the experiments, those parameters must be defined by the user. The meaning of each column is explained next:
– Column A: The minimum number of instances required to start the analysis of the temporary buffer.
– Column B: The minimum number of instances to consider a MC representative.
– Column C: The Forgetting Factor, this number indicates how long an instance is kept in the temporary buffer. The forgetting process of the temporary buffer is performed by subtracting the Time Stamp of the instances in the temporary buffer to the current Time Stamp, if the result is greater that the Forgetting Factor the instance is discarded from the temporary buffer.
– Column D: The number of MC’s to be generated from the temporary buffer.
– Column E: The number of MC’s generated for each class in the training phase, this number refers to the k value for k-means.
Table 2 Example of initial training data
| Number of instance | Attribute 1 | Attribute 2 | Attribute 3 | Original Class |
| 1 | 0.85 | Yes | 5 | Class B |
| 2 | 0.36 | Yes | 10 | Class A |
| 3 | 0.97 | No | 7 | Class B |
| 4 | 0.14 | Yes | 15 | Class A |
| 5 | ’?’ | No | 6 | Class B |
The first experiment carried out was KDD Cup 1999 with its 10 majority classes. As the data set has
Our proposal reaches in all the data chunks the 100% of the accuracy while Hoeffding Tree sometimes reaches the
Figure 10 corresponds to the second one experiment, which was Statlog (Shuttle), having
In Table 4 we observe the average accuracies of the whole experiments, for the second time our proposal manages to obtain the highest average accuracy compared to Näives Bayes and Hoeffding Tree. Here concept-drift was identified in 3 classes.
Table 4 Accuracies averages
| Using NACOD | Näive Bayes | Hoeffding Tree |
| 100% | 99.27% | 99.81% |
| 99.01% | 90.02% | 97.81% |
| 70.44% | 59.59% | 67.78% |
The aim here is to obtain an accuracy of
As in the previous experiment, our proposal also identified concept-drift in
5 Conclusion and Future Work
5.1 Conclusions
In this work an algorithm for classification of data streams using a classifier with an associative approach was designed, implemented and tested. This proposal combines a clustering algorithm, the Naïve Associative Classifier for Online Data (NACOD), a method to detected concept-drift and an updating process to incorporate the new discovered knowledge.
An extensive documentary research was carried out to study the issues related to the processing of massive data streams that change dynamically. As part of the study of current topics in data stream learning, a study of available real data sets was performed, in order to select those that were appropriate to perform the tests of the proposed algorithm. This led us to the selection of only 3 data sets for our tests.
The proposed solution was developed with the aim of classify, but also addresses concept-drift and noisy data. Our proposal beats the other classifiers in the evaluation measure. Because of the design of the solution, it is our belief that it is also capable of handling recurrent concepts, but given the scope, tests to verify this affirmation were not performed.
5.2 Future Work
The proposal presented in this paper fulfilled the main objective of classifying into data streams but several issues are still open to improvements to extend the scope of this proposal. Among them, we can mention the following:
– Use a different clustering algorithm to create the MC’s. A clustering algorithm specially design for data stream, such as CluStream, could improve the results.
– Test the proposed solution with synthetic generated data sets.
– Design experiments to test the ability of the algorithm to handle recurring concepts and concept evolution.
