Predict Wins and Losses with Sci-kit Learn Decision Trees and SMS
Tags: pandas, scikitlearn, twilio, machinelearning
Published: 2020-05-06 23:33:40

---
title: Predict Wins and Losses with Sci-kit Learn Decision Trees and SMS
published: true
description: Can machine learning predict whether or not a sports team wins or loses? What statistics best predict whether that outcome? This post will go over how to use Sci-kit Learn decision trees, Pandas, and Twilio Programmable SMS to answer those questions.
tags: pandas, scikitlearn, twilio, machinelearning
---
![header img](https://twilio-cms-prod.s3.amazonaws.com/images/_RkZi13-H3zteJ4X2sIkzZ9RIx-fZ7qvbmfs2SafQhLyH.width-1616.png)

This blog post was written for [Twilio](https://www.twilio.com/) and [originally published on the Twilio blog](https://www.twilio.com/blog/predict-wins-losses-scikitlearn-sms).

I've been missing professional sports which are canceled for the time being. It got me thinking -- can we predict a basketball game outcome with machine learning? What stats of an individual player would best predict a team's win or a loss? This post will try to answer those questions (using [Klay Thompson stats](https://www.basketball-reference.com/players/t/thompkl01.html)) with [Pandas](https://pandas.pydata.org/) and [sci-kit learn](https://scikit-learn.org/) based on user input via a Twilio SMS containing a certain basketball statistic. With SMS user input, it is easy to compare which statistics are better suited for predicting whether the team won or loss because of that individual player.
![klay hand gif](https://twilio-cms-prod.s3.amazonaws.com/original_images/HKBRj-vjDMr9Drr93gfk0PK8upBi19OEw_e2kh0lG8mtuR8WyY6R9dA5FdT17_n_tFbVJjq5DGl86z)

### Soo...Decision Trees
They make a training model to predict the outcome, class, or value of a target variable based on rules the model learns from input data. They model decisions in a tree-like manner drawn upside-down with the root at the top. Below is a [weather decision tree from Juniata College](http://jcsites.juniata.edu/faculty/rhodes/ida/decisionTrees.html) deducing whether it is sunny, overcast, or raining.
![decision tree graphic](https://twilio-cms-prod.s3.amazonaws.com/images/HA649YZPsZyYoVDmbI-uECLpUk6pmZBTigQQG7uYYtWRdI.width-500.png)

They are often used for both classification (output is categorical and discrete) and regression (result is numerical and continuous) in machine learning. Decision tree algorithms are sometimes called CART (Classification and Regression Trees).

Some vocabulary often used when talking about decision trees:
1. <em>Entropy</em>, or the amount of variance in the data. If a bag of M&M's had only brown ones it would have low entropy; if a bag of M&M’s had brown, red, green, yellow, and orange, the entropy would be high.
2. <em>Root node</em>: the top of the tree and where the training data or sample begins before being split into two or more sets.
3. <em>Split</em>: the tree splits or branches off into child nodes based on an attribute.
4. <em>Leaf</em> or <em>terminal node</em>: a node that does not have child nodes.

In short, decision trees identify ways to split a data sample according to different conditions. They can predict future outcomes and assign probabilities to those outcomes, determine whether or not to continue an experiment, and more. This blog post will show you how to code decision trees with some popular data mining and machine learning Python libraries.

### Prerequisites
1. A Twilio account - [sign up for a free one here and receive an extra $10 if you upgrade through this link](http://www.twilio.com/referral/iHsJ5D)
2. A Twilio phone number with SMS capabilities - [configure one here](https://www.twilio.com/console/phone-numbers/search)
3.[Set up your Python and Flask developer environment](https://www.twilio.com/docs/usage/tutorials/how-to-set-up-your-python-and-flask-development-environment). Make sure you have [Python 3](https://www.python.org/downloads/) downloaded as well as [ngrok](https://ngrok.com/).

### Setup your Python Project and a Twilio Number
Activate a virtual environment in Python 3 and on the command line run the following to install our dependencies:
```bash
pip3 install sklearn pandas numpy flask twilio
```
Your Flask app will need to be visible from the web so Twilio can send requests to it. [Ngrok](https://ngrok.com/) lets us do this. With it installed, run the following command in your terminal in the directory your code is in in a new terminal tab: `ngrok http 5000`.
![ngrok url img](https://twilio-cms-prod.s3.amazonaws.com/images/h4udHoNeKLobyRDRTVzontpp26Zzo0hofjQ_E6qt-MoUK.width-1000.png)

You should see the screen above. Grab the https://  ngrok forwarding URL to configure your Twilio number in your [phone number console](https://www.twilio.com/console/phone-numbers/incoming). If you don't have a Twilio number yet, go to the [Phone Numbers section of your Twilio Console](https://www.twilio.com/console/phone-numbers) and search for a phone number in your country and region, making sure the SMS checkbox is ticked.
![buy a number](https://twilio-cms-prod.s3.amazonaws.com/images/dyk1GnLU45wb9o5_7LZ6s8Z04XFnSdQc4gG1GeztKS96o.width-1000.png)

In the `Messaging` section of your purchased number, in the `A Message Comes In` section, set the `Webhook` to be your ngrok https URL, appending `/sms` to the URL. Hit Save.

![messaging section of console](https://twilio-cms-prod.s3.amazonaws.com/images/hC0TaatCi2gLrC_fgfeCXxi_h9YklvdYqZESQWGepYXPz.width-1000.png)

### Access and Clean Data
As with most ML tasks, we need good data. I like the basketball team the [Golden State Warriors](https://www.nba.com/warriors/), so I wanted data related to them. I found game-by-game data for individual players. Klay Thompson played quite a few games (and is one half of the [Splash Brothers](https://en.wikipedia.org/wiki/Splash_Brothers)) so I used this [game-by-game stats from him for the 2018-2019 season](https://www.basketball-reference.com/players/t/thompkl01/gamelog/2019/):
![klay stats](https://twilio-cms-prod.s3.amazonaws.com/images/e8_sl-BA07ksllICeQtqEx3lS-_uNMDrcJm6cCyhSCiiH.width-1000.png)

If you are not interested in the process I used to export and clean up this data you can skip the rest of this section and [use this file containing the clean data from GitHub](https://raw.githubusercontent.com/elizabethsiegle/sklearn-predict-warriors-win-from-klaythompson-stats/master/klay_season_stats.csv).
1. Hover over <em>Share & More</em>
2. Click <em>Modify & Share Table</em>
![modify and share table img](https://twilio-cms-prod.s3.amazonaws.com/images/-0x7PifuNnOqAVVwEk7VsD4ycnVboxNXqT5s9qvJS8vZS.width-1000.png)
3. Select <em>comma-separated</em> under <em>Share or get code for table as</em>:
4. Copy-and-paste this data to a .csv file (my folder is called <em>klay</em> and the file <em>klay_season_stats.csv</em>).

Some of the columns have empty labels. Hand-edit the CSV file so that there is a H-A column header label (for <em>home</em> or <em>away</em>) at the top in-between <em>Tm</em> and <em>Opp</e,> and a Win-Loss column header label in-between <em>Opp</em> and <em>GS</em>. Search and replace "W (" with "W", "L (" with "L", and ")" with ",". ("W" represents a win and "L" represents a loss.) After <em>Win-Loss</em>, add a header WLD that has the difference in scores which follow the <em>Win-Loss</em>. That should end up looking like this:
![final data](https://twilio-cms-prod.s3.amazonaws.com/images/xib3-3Y_zS8TvPaz0c366spCqwXTphuqZ-jsHORvMKeCm.width-1000.png)
The [updated and cleaned data can be found here on GitHub](https://raw.githubusercontent.com/elizabethsiegle/sklearn-predict-warriors-win-from-klaythompson-stats/master/klay_season_stats.csv).

### Read and Manipulate Data with Pandas
At the top of your Python file (mine is called `predict_with_klay.py`), import the required libraries.
```python
from sklearn.metrics import confusion_matrix, classification_report,  accuracy_score
from sklearn import tree, preprocessing
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.tree import DecisionTreeClassifier
import pandas as pd
import numpy as np
from twilio.twiml.messaging_response import MessagingResponse
from flask import Flask, request
```
Then save the `csv` file as a variable and set the columns we want to consider for this blog post.
```python
csv = 'klay_season_stats.csv'
cols = ['FG', 'FGA', 'FG%', '3P', '3PA', '3P%',
        'FT', 'FTA', 'FT%', 'ORB', 'DRB', 'TRB']
```
Add the following code below to make a function that takes in the CSV file and an incoming SMS message that contains a statistic like FG, FGA, etc.
```python
def setup_df(file, msg):
    df = pd.read_csv(csv)
    print(df.shape) # (83, 31) = 83 records/rows, 31 attributes/columns
    df = df[['H-A', 'FG%', '3P%', 'PTS', msg, 'WL']] #last item = classifier

    #remove row with missing values
    df = df.fillna(0)

    #convert strings to integers
    df['WL'] = df['WL'].map({'W': 1, 'L': 0})
    df['H-A'] = df['H-A'].map({'H': 1, 'A': 0})

    #access row with Did Not Dress, etc, replacing Did Not Dress, Inactive with 0's
    #0 axis = rows, 1 axis = columns
    m1 = df.eq('Did Not Dress').any(axis=1)
    m2 = df.eq('Did Not Dress').any(axis=0)
    m3 = df.eq('Inactive').any(axis=1)
    m4 = df.eq('Inactive').any(axis=0)
    
    df.loc[m1] = 0
    df.loc[:, m2] = 0
    df.loc[m3] = 0
    df.loc[:, m4] = 0
    return df
```
The function makes a [Pandas data frame](https://pandas.pydata.org/pandas-docs/stable/dsintro.html) `df`, a 2D tabular data structure with labeled axes for rows and columns to make manipulating your data easier. Then we set `df` to be only the classes we want to examine (H-A, FG%, 3P%, PTS, and the user's input.) The last item in that array, whether the Warriors won or loss, is the classifier, or what we want to predict. We remove any rows with missing values, convert strings to integers: Any <em>Win</em> will be converted to 1 and <em>Loss</em> to 0. Then we look for games where Klay Thompson did not dress or was inactive. We still want to see whether or not the Warriors won with him gone, so we convert those values to zero. This isn't always the best method as NaN values could still have significance in being absent from the dataset. Introducing zeros could skew the data, but in this case zero is an acceptable value because Klay did have zero three-pointers, field goals, etc. in those games.

Now we clean the dataset of NaN, Inf, and missing cells for skewed datasets, keeping the DataFrame with valid entries in the same variable.
```python
def clean_dataset(df):
    df.dropna(inplace=True)
    indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(1)
    return df[indices_to_keep].astype(np.float64)
```

### Train, Make, Evaluate Predictions
First we need a Flask application to get the inbound text message. We define a `/sms` endpoint that listens to POST requests. We then check that the message is in the `cols` list of statistics we're considering. If so, we call the `setup_df` method to make our data frame data structure with the CSV file and inbound SMS before passing that `df` to clean_dataset to clean it into data we can use. A variable `X` holds every column from our dataset except the "WL" column, which is the label and a `y` variable contains the classification class values from the "WL" column (whether the Warriors won or lost).

```python
app = Flask(__name__)
@app.route("/sms", methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body']
    if inb_msg in cols:
        df = setup_df(csv, inb_msg)
        clean_dataset(df)
        print(df.head())  # first 5 records of dataset  
        print(df)  # only data we're looking at with user input (also H-A, FG)
        X = df.drop('WL', axis=1)
        y = df['WL']
```
Then we use Scikit-Learn's `model_selection` library's `train_test_split` method to split the data into training and testing sets. The training set is used to check that the algorithm recognizes patterns in the data and the testing set is used to see how well the algorithm can predict new answers based on its training.

`test_size` sets the ratio of the test set used to split-up 20% of the data into the test set and 80% for the training set.
```python
X_train, X_test, y_train, y_test = train_test_split(
            X, y, test_size=0.20)
```
The `DecisionTreeClassifier` class provides an algorithm to train on the testing and training data and make predictions with. The `fit` method trains the algorithm on the training data passed as a parameter to it.

```python
model = tree.DecisionTreeClassifier()
        model.fit(X_train, y_train)
```
Lastly we make our predictions:
```python
y_pred = model.predict(X_test)
```
Sci-kit Learn provides some handy-dandy methods to evaluate those predictions. [classification_report](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) provides precision (of the positive classes correctly predicted, the number that is actually positive), recall (number predicted correctly--the higher, the better), F1 (harmonic mean between precision and recall), and support score (number of occurrences of the given class in the dataset) for the model. The following images are when "ORB" for offensive rebounds are considered as a statistic.
```python 
print(classification_report(y_test, y_pred))
```
![classification report img](https://twilio-cms-prod.s3.amazonaws.com/images/mTyODyMcI7OTN0JIW_LR0UpmhzS5yoTY-EstFMFP6nF_b.width-1000.png)

Then the `cross_val_score` method performs [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)), testing the result by setting the `scoring` parameter to `accuracy`.

```python
accuracy = np.mean(cross_val_score(model, X_test, y_test, scoring='accuracy')) * 100
        print("Accuracy: {}%".format(accuracy))
```
![accuracy printed to console](https://twilio-cms-prod.s3.amazonaws.com/images/vQ5DTqFFNOA9fmIsqmPkTv0fIc0S1KIamBbuBJT6zUnLdv.width-500.png)

Your accuracy will be different, but this accuracy is pretty good. Anything over 50% means the model is better than random. It means that the decision tree model has correctly classified 77% of the sample correctly as the Warriors winning or losing based on the user input. (Again, this accuracy includes offensive rebounds.)

Lastly, the [Confusion Matrix](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) returns the correct and incorrect classifications of our decision tree model, or the predicted outcomes compared to the actual outcomes.

```python
print('confusion matrix {}'.format(pd.DataFrame(
            confusion_matrix(y_test, y_pred),
            columns=['Predicted Loss', 'Predicted Win'],
            index=['True Loss', 'True Win']
        )))
```
The confusion matrix returns four values without normalization: True Positive, True Negative, False Positive, and False Negative. They are in the locations below.

| True Positive  | False Positive |
|----------------|----------------|
| False Negative | True Negative  |

Our decision tree model returns no true positives and no false negatives, and four false positives and thirteen true negatives.
![confusion matrix printed to console](https://twilio-cms-prod.s3.amazonaws.com/images/bvj8cURWhGuBKpoMUkAsvSU7fNAl-UHzju2-dT_KcVAwz.width-1000.png)

We then return these performance metrics in an outbound message.
```python
msg = 'The Decision Tree model correctly predicted if the Warriors won or lost based on some of Klay\'s stats {:.2f} % of the time.\n The confusion matrix looks like this:\n {}'.format(
            accuracy, confusion_matrix(y_test, y_pred))
```
If the user does not send a statistic in our `cols` list, we tell them what messages they should send and then return our message.
```python
else:
        msg = "Send a message according to Klay Thompson's stats columns: {}".format(cols)
    resp.message(msg)
    return str(resp)
```
That was a lot of code! The complete code for that function is below.
```python
app = Flask(__name__)
@app.route("/sms", methods=['GET', 'POST'])
def sms():
    resp = MessagingResponse()
    inb_msg = request.form['Body']
    if inb_msg in cols:
        df = setup_df(csv, inb_msg)
        clean_dataset(df)
        print(df.head())  # first 5 records of dataset  
        print(df)  # only data we're looking at with user input (also H-A, FG)
        X = df.drop('WL', axis=1)
        y = df['WL']
        # specifies ratio of test set, used to split up 20% of the data into test set and 80% for training
        X_train, X_test, y_train, y_test = train_test_split(
            X, y, test_size=0.20)  # random_state=1
        model = tree.DecisionTreeClassifier()
        # print(model)
        model.fit(X_train, y_train)
        y_pred = model.predict(X_test)
        print(classification_report(y_test, y_pred))
        accuracy = np.mean(cross_val_score(model, X_test, y_test, scoring='accuracy')) * 100
        print("Accuracy: {}%".format(accuracy))

        print('confusion matrix {}'.format(pd.DataFrame(
            confusion_matrix(y_test, y_pred),
            columns=['Predicted Loss', 'Predicted Win'],
            index=['True Loss', 'True Win']
        )))
        # distributions for each group
        msg = 'accuracy score: {}\n confusion matrix: {}\n distributions for predicted group: {}\n'.format(
            accuracy, confusion_matrix(y_test, y_pred), y_test.value_counts(normalize=True))
    else:
        msg = "Send a message according to Klay Thompson's stats columns: {}".format(cols)
    resp.message(msg)
    return str(resp)
```
If you run
```
export FLASK_APP=predict_with_klay 

export FLASK_ENV=development

flask run
```
on a Mac (if you're on Windows, replace `export` with `set`) from the command line and text your Twilio number a statistic, you should get a response like this:
![texts](https://twilio-cms-prod.s3.amazonaws.com/images/Q4Bmtbxdn3k7lX_3bv5V8Fiw_M2Qu6PoIVgKUhzpUDPV8.width-1000.png)

The complete code can be found [here on GitHub](https://github.com/elizabethsiegle/sklearn-predict-warriors-win-from-klaythompson-stats). It seems that Klay Thompson's free-throw percentage is better than three-point percentage in predicting whether or not the Warriors won that game, and field goal percentage is not good at predicting whether they won or lost.

### What will you Predict Next?
![tip of iceberg gif](https://twilio-cms-prod.s3.amazonaws.com/original_images/4uaCAlL85LL4DGw-zz9beqcjjkBEgDN0Q7JRY23I3_mhYgGakQ0tlqwblpm7AhHy4gHnegnKEpGJLW)

This is just the tip of the iceberg for developing with decision trees, Pandas, and Sci-kit Learn. You can also utilize specific algorithms like [random forests](https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm) and [k-neighbors](https://www.kdnuggets.com/2020/04/introduction-k-nearest-neighbour-algorithm-using-examples.html), make a [Naive Bayes classifier](https://scikit-learn.org/stable/modules/naive_bayes.html), better optimize decision tree performance, use non-sports data, and more! Let me know in the comments or online what you're building with machine learning.
- Twitter: [@lizziepika](http://twitter.com/lizziepika)
- GitHub: [elizabethsiegle](http://github.com/elizabethsiegle)
- Email: lsiegle@twilio.com