If sampling is done on the dataset, this can be done by explicitly specifying the size of the sample, or on probability, and also a filter can be used, in the case when the parts of the dataset are not to be represented proportionally in the sample, rather a given subset of the original dataset is necessary for the process. For example, filtering for the records belonging to a given time on every day can be done as follows: Figure 4.5. Preferences for dataset sampling Figure 4.6. Preferences for dataset filtering Interpretation of the results performing the aggregation or sampling, the received dataset will only consist of the aggregate values emerging as a result of the specified operations, or the records that fulfil the specified conditions, respectively: Figure 4.7. Resulting dataset after dataset sampling Figure 4.8. Resulting dataset after dataset filtering Video Workflow preproc_exp2.rmp Keywords aggregation summation sampling data filtering Operators Aggregate Filter Examples Multiply Read CSV Sample 3. Creating and filtering attributes Description The process shows, using The Insurance Company Benchmark (COIL 2000) dataset, how new, computed attributes can be created based on existing data, in case the attributes are not appropriate in their original form,
Operators Apply Model Normalize Performance (Classification) Read CSV Support Vector Machine (LibSVM) 13. Using an SVM for solving a regression problem Description The process demonstrates how to use an SVM for solving a regression problem. In this experiment RBF kernel SVMs are trained on the Concrete Compressive Strength data set while the value of the parameter gamma of the RBF kernel is changed. To obtain comparable results the value of the parameter C is fixed to 10. The average RMS error from 10-fold cross-validation is determined for each SVM. As a result, the gamma value yielding the best average RMS error will be returned. Using this value for the parameter gamma an RBF kernel SVM is trained on the entire data set that is referred to as the “optimal RBF kernel SVM” below. Input Concrete Compressive Strength [] [] Output Figure 8.26. The optimal of the gamma parameter for the RBF kernel SVM. Figure 8.27. The average RMS error of the RBF kernel SVM obtained from 10-fold cross-validation against the value of the parameter gamma, where the horizontal axis is scaled logarithmically. Figure 8.28. The kernel of the optimal RBF kernel SVM. Figure 8.29. Predictions provided by the optimal RBF kernel SVM against the values of the observed values of the dependent variable. Interpretation of the results The first figure shows that the best average RMS error is achieved when the value of the parameter gamma is = 0.25. The third figure shows that the average RMS error decreases with the increasing value of the parameter gamma until it reaches its minimum. However, further increase of the value of the parameter gamma results in the degradation of the performance, i.e., model overfitting occurs. Video
Then the points are assigned to these groups. Interpretation of the results It can be seen that indeed, the method has created a larger number of clusters, but due to this, the central clusters can be separated better from each other. Video Workflow clust_exp5.rmp Keywords Divisive method divisive hierarchical clustering cluster analysis Operators k-Means Read CSV Top Down Clustering
Chapter 13. Anomaly detection 1. Searching for outliers Description The workflow, using the Wisconsin Diagnostic Breast Cancer (WDBC) dataset, shows how to find outliers based on the distances measured between the data. This can be done either by measuring their distance from their k nearest neighbours, or by checking whether their distance from some data object is above a given threshold. The definition of an outlier is relative, it can always be defined in comparison with the distances between the data objects. Thus if the distances between the data objects are basically great, a high threshold has to be set for outliers. Input Wisconsin Diagnostic Breast Cancer (WDBC) [] Output It can be seen that outliers can be filtered out using the appropriate settings. As for example even differences that range in the hundreds occur between the individual values of the represented attribute area, thus this result can be obtained by setting the threshold for the Euclidean distance to the value 500. Figure 13.1. Graphic representation of the possible outliers Interpretation of the results Note that due to the existing great distances between the data objects, the number of outliers detected will only decrease to its true level if the threshold is incremented up to 500, and under a certain value, way too many data objects would be identified as outliers. Figure 13.2. The number of outliers detected as the distance limit grows Video Workflow anomaly_exp1.rmp Keywords outliers data preprocessing data cleansing Operators Detect Outlier (Densities) Detect Outlier (Distances)
Chapter 15. Preprocessing 1. Constructing metadata and automatic variable selection Description The process illustrates, by using the Spambase dataset, how to generate the metadata of a dataset by the DMDB operator, then how automatic variable selection can be obtained by the Variable Selection operator. The Spambase dataset contains 58 attributes, one of which is the binary target. In order to visualize a dataset, it may be necessary to determine the most important input attributes which can be used in the graphical representation. Spambase [] Output The DMDB operator produces such metadata (descriptive statistics) as mean, variance, minimum, maximum, skewness, and kurtosis. In case of discrete attributes these are complemented by the mode. Figure 15.1. Metadata produced by the DMDB operator The default settings of the Variable Selection operator are applied except that the minimum R-square is increased in to filter the unnecessary attributes. Figure 15.2. The settings of Variable Selection operator The result on the one hand will be a list contains the decision about the variables, i.e., whether it remains or not in the data mining process, on the other hand, a few graphs of the importance of the variables. Figure 15.3. List of variables after the selection Figure 15.4. Sequential R-square plot In view of the important variables a number of graphical tools of the Enterprise Miner ™ can be used to display the records. Figure 15.5. The binary target variables in a function of the two most important input attributes after the variable selection Interpretation of the results
The main result of principal component analysis is the principal component coordinates of individual records, which can be used in the further data analysis and visualization. Figure 15.8. Scatterplit of the Iris dataset using the first two principal components Interpretation of the results The experiment shows that how we can display high-dimensional data sets and perform dimension reduction. In our experiment, the original 4-dimensional data set that can not be displayed using the standard scatterplot, is managed to reduce to 2 dimensions such that the 95 percent of the information contained in the data is preserved. Video Workflow sas_preproc_exp2.xml Keywords principal components analysis (PCA) parallel axis Operators Data Source Graph Explore Principal Components 3. Replacement and imputation Description In this experiment, we demonstrate by the help of the Congressional Voting Records dataset how to modify the values of attributes by the Replacement operator and then how to impute the missing value by the Impute operator. The replacement of missing values each variable can be carried out independently of the others and to interact with the target variable by fitting a decision tree. Input Congressional Voting Records [] Output By the Replacement operator we can set the substitution of discrete and continuous variables separately. Figure 15.9. The replacement wizard