Elastic Data Frame - Classification Analysis HandsOn
Tags: machinelearning, elasticsearch, tutorial, beginners
Published: 2021-08-02 14:36:28

| [Menu](https://dev.to/priscilla_parodi/a-guide-to-machine-learning-ai-with-the-elastic-stack-1dkl) | [Next Post: Elastic Data Frame - Inference Processor HandsOn](https://dev.to/priscilla_parodi/elastic-data-frame-inference-processor-handson-3392) |

> *Note: This HandsOn assumes that you have already followed the step-by-step Setup of your Elastic Cloud account and added the Samples available there to replicate the analysis mentioned here. If not, please, [follow the steps mentioned there](https://dev.to/priscilla_parodi/handson-setup-elastic-cloud-4j8p).*

For the [Classification Analysis](https://dev.to/priscilla_parodi/elastic-data-frame-classification-analysis-1b3f) we will use the `Flights Sample` dataset.

Kibana>Analytics>Dashboard>Search for and click on:`[Flights] Global Flight Dashboard`.

If your Dashboard is empty it is possibly because the time interval is wrong, select `Last 1 year` where it usually is `Last 24 hours` as in the image below, and the Dashboard will automatically reload.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fhos5tvjsfrpx2vhug1r.png)

Feel free to explore the data to understand what information we don't yet have and what might be useful to analyze. Remembering that this is just a simulation, this is not real data and the available data is limited.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6yp89ih8n9gbbf7it929.png)

An essential aspect of airport management is to closely monitor and anticipate possible flight delays and cancellations in order to prevent them whenever possible.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bharxaortacebtuiekvn.png)

So, let's train a model to predict delays based on input data!

Machine Learning>Data Frame Analytics>+Create Job and search for and click on `[Flights] Flight Log`.

Select `Classification` and in the Dependent variable, variable that we will predict the class, add the boolean variable `FlightDelay`, as in the image below.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vzdjzv2wzprwyitlc430.png)

And then you can choose what to include or not in the analysis. Of course, the more useful data you analyze, the better the result and effectiveness, but remember not to exclude data that you are not sure about the need, because some fields that we judge to have no relation to the variable we are classifying may have some implicit relationship.

In this case, we are not going to include in the analysis data that can influence the algorithm to know the answer only with the value of directly correlated fields, I want the algorithm to use the other fields.

I will not include/select `FlightDelayMin` and `FlightDelayType`. We will not remove the `Canceled` boolean variable from the analysis because a delayed flight will not necessarily be canceled and we will not remove `AvgTicketPrice` either, maybe there is a relationship between price and delay, we don't want to exclude possibly useful fields, same thing for `FlightNum`.

Select the fields you want, as in the image below:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cg016b90il04th843h33.png)

For Scatterplot Matrix, feel free to add data you want to see the relationship.

And finally add the percentage of documents destined for training and testing. 80 means that 80% of the data will be used for training and 20% for tests performed after training to test the effectiveness of the model in predicting correct results based on existing documents. But let's use 85% for training as we don't have much data. Click Continue.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vffe12o8beqb5glvh846.png)

In step 2, we will not change any settings at this time, but feel free to customize and click Continue.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xmsgucan3mvn3ieol8sq.png)

In step 3 add the name of the Job ID you want, an index pattern will be created. I will use the name: `delay-prediction`. Click Continue.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4pq4g068uyj75ngagmqv.png)

Your Validation should look like the image below:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/445fwrdmen2zaopaiseo.png)

Click continue and then click Create.

8 phases need to be loaded and you will be able to follow the progress. 

When your screen looks like the image below, click View Results.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5pmmy6l98czn089vtqwm.png)

And these will be the results displayed in Kibana:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xiwiv8aap0l2oplckg0q.png)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3l09ke2xl0472se4qxdj.png)

The recall value is calculated as the number of correct predictions (where the actual label matches the predicted label) divided by the sum of all the other predicted labels in that row.

And as you can see in the images above, [the result was satisfactory for training and testing](https://dev.to/priscilla_parodi/elastic-data-frame-classification-analysis-1b3f). It is rare to get 100% in some row of the normalized confusion matrix. But the more data for training and testing, the better. On that same page you can see the variables that most affected prediction and the Scatterplot matrix, which you can customize, as well as the results with prediction data.

But let's check the documents in our new index pattern, Kibana>Analytics>Discover and select the index pattern `delay-prediction` as in the image below:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9jb9whbcqd3lbw5tdttd.png)

Expanding a document we can see the result of the analysis, the `FlightDelay` field is our original field and the `ml.FlightDelay_prediction` field is the field that the model is predicting, in this case both fields have the same "false" value, which means that the prediction is correct.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hiomyr1kndm0t2uszjh2.png)

You can also verify your model data by following [these steps](https://dev.to/priscilla_parodi/trained-models-for-supervised-learning-154n), in addition to accessing [APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-df-analytics-apis.html), and [using your model against incoming data](https://dev.to/priscilla_parodi/elastic-data-frame-inference-processor-handson-3392).

| [Menu](https://dev.to/priscilla_parodi/a-guide-to-machine-learning-ai-with-the-elastic-stack-1dkl) | [Next Post: Elastic Data Frame - Inference Processor HandsOn](https://dev.to/priscilla_parodi/elastic-data-frame-inference-processor-handson-3392) |

>*This post is part of a series that covers [Artificial Intelligence with a focus on Elastic's (Creators of Elasticsearch) Machine Learning solution](https://dev.to/priscilla_parodi/a-guide-to-machine-learning-ai-with-the-elastic-stack-1dkl), aiming to introduce and exemplify the possibilities and options available, in addition to addressing the context and usability.*