Homemade Machine Learning in Python
Tags: python, programming, machinelearning, ai
Published: 2018-12-21 06:41:01

---
title: Homemade Machine Learning in Python
published: true
description: Python examples of popular machine learning algorithms with interactive Jupyter demos and math being explained
tags: python, programming, machinelearning, ai
---


<center><img src="https://thepracticaldev.s3.amazonaws.com/i/clw393m8289hal97m0x8.png" />
<i><small><a href="https://vas3k.ru/blog/machine_learning/">Image source</a></small></i></center>

<div class="section-inner sectionLayout--insetColumn"><p name="fe39" id="fe39" class="graf graf--p graf-after--figure">I’ve recently launched <a href="https://github.com/trekhleb/homemade-machine-learning" data-href="https://github.com/trekhleb/homemade-machine-learning" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank"><strong class="markup--strong markup--p-strong">Homemade Machine Learning</strong></a> repository that contains examples of popular machine learning algorithms and approaches (like <em class="markup--em markup--p-em">linear/logistic regressions, K-Means clustering, neural networks</em>) implemented in <strong class="markup--strong markup--p-strong">Python</strong> with mathematics behind them being explained. Each algorithm has interactive <strong class="markup--strong markup--p-strong">Jupyter Notebook</strong> demo that allows you to play with training data, algorithms configurations and immediately see the results, charts and predictions <strong class="markup--strong markup--p-strong">right in your browser</strong>. In most cases the explanations are based on <a href="https://www.coursera.org/learn/machine-learning" data-href="https://www.coursera.org/learn/machine-learning" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">this great machine learning course</a> by <a href="https://medium.com/@andrewng" data-href="https://medium.com/@andrewng" data-anchor-type="2" data-user-id="592ce2a67248" data-action-value="592ce2a67248" data-action="show-user-card" data-action-type="hover" class="markup--user markup--p-user" target="_blank">Andrew Ng</a>.</p><p name="c5f5" id="c5f5" class="graf graf--p graf-after--p">The purpose of the repository was <em class="markup--em markup--p-em">not</em> to implement machine learning algorithms by using 3rd party library “one-liners” <em class="markup--em markup--p-em">but </em>rather to practice implementing these algorithms from scratch and get better understanding of the mathematics behind each algorithm. That’s why all algorithms implementations are called “homemade”.</p><p name="5d0e" id="5d0e" class="graf graf--p graf-after--p">The main Python libraries that are used there are <a href="http://www.numpy.org/" data-href="http://www.numpy.org/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">NumPy</a> and <a href="https://pandas.pydata.org/" data-href="https://pandas.pydata.org/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">Pandas</a>. These two are used for efficient matrix operations and for loading/parsing CSV datasets. When it comes to <a href="http://jupyter.org/" data-href="http://jupyter.org/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">Jupyter Notebook</a> demos then such libraries as <a href="https://matplotlib.org/" data-href="https://matplotlib.org/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">Matplotlib</a> and <a href="https://plot.ly/" data-href="https://plot.ly/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">Plotly</a> are being used for data visualizations.</p><p name="a3d1" id="a3d1" class="graf graf--p graf-after--p">Currently the following topics have been covered:</p><h4 name="e765" id="e765" class="graf graf--h4 graf-after--p">Regression: Linear Regression</h4><p name="1b45" id="1b45" class="graf graf--p graf-after--h4">In regression problems we do real value predictions. Basically we try to draw a line/plane/n-dimensional plane along the training examples.</p><p name="2928" id="2928" class="graf graf--p graf-after--p"><em class="markup--em markup--p-em">Usage examples: stock price forecast, sales analysis, dependency of any number, etc.</em></p><ul class="postList"><li name="a2ff" id="a2ff" class="graf graf--li graf-after--p">📗 <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/linear_regression" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/linear_regression" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Linear Regression Math</a>&#8202;—&#8202;theory and links for further readings</li><li name="6a53" id="6a53" class="graf graf--li graf-after--li">⚙️ <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/linear_regression/linear_regression.py" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/linear_regression/linear_regression.py" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Linear Regression Implementation Example</a></li><li name="a82a" id="a82a" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/univariate_linear_regression_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/univariate_linear_regression_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Univariate Linear Regression</a>&#8202;—&#8202;predict <code class="markup--code markup--li-code">country happiness</code> score by <code class="markup--code markup--li-code">economy GDP</code></li><li name="a0e9" id="a0e9" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/multivariate_linear_regression_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/multivariate_linear_regression_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Multivariate Linear Regression</a>&#8202;—&#8202;predict <code class="markup--code markup--li-code">country happiness</code> score by <code class="markup--code markup--li-code">economy GDP</code> and <code class="markup--code markup--li-code">freedom index</code></li><li name="c6b6" id="c6b6" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/non_linear_regression_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/linear_regression/non_linear_regression_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Non-linear Regression</a>&#8202;—&#8202;use linear regression with <em class="markup--em markup--li-em">polynomial</em> and <em class="markup--em markup--li-em">sinusoid</em> features to predict non-linear dependencies.</li></ul><h4 name="aa9d" id="aa9d" class="graf graf--h4 graf-after--li">Classification: Logistic Regression</h4><p name="7f81" id="7f81" class="graf graf--p graf-after--h4">In classification problems we split input examples by certain characteristic.</p><p name="fd83" id="fd83" class="graf graf--p graf-after--p"><em class="markup--em markup--p-em">Usage examples: spam-filters, language detection, finding similar documents, handwritten letters recognition, etc.</em></p><ul class="postList"><li name="e1a3" id="e1a3" class="graf graf--li graf-after--p">📗 <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Logistic Regression Math</a>&#8202;—&#8202;theory and links for further readings</li><li name="ae39" id="ae39" class="graf graf--li graf-after--li">⚙️ <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression/logistic_regression.py" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/logistic_regression/logistic_regression.py" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Logistic Regression Implementation Example</a></li><li name="bff3" id="bff3" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/logistic_regression_with_linear_boundary_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/logistic_regression_with_linear_boundary_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Logistic Regression (Linear Boundary)</a>&#8202;—&#8202;predict Iris flower <code class="markup--code markup--li-code">class</code> based on <code class="markup--code markup--li-code">petal_length</code> and <code class="markup--code markup--li-code">petal_width</code></li><li name="2df2" id="2df2" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/logistic_regression_with_non_linear_boundary_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/logistic_regression_with_non_linear_boundary_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Logistic Regression (Non-Linear Boundary)</a>&#8202;—&#8202;predict microchip <code class="markup--code markup--li-code">validity</code> based on <code class="markup--code markup--li-code">param_1</code> and <code class="markup--code markup--li-code">param_2</code></li><li name="ce5a" id="ce5a" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/multivariate_logistic_regression_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/logistic_regression/multivariate_logistic_regression_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Multivariate Logistic Regression</a>&#8202;—&#8202;recognize handwritten digits from <code class="markup--code markup--li-code">28x28</code> pixel images.</li></ul><h4 name="0c61" id="0c61" class="graf graf--h4 graf-after--li">Clustering: K-means Algorithm</h4><p name="c887" id="c887" class="graf graf--p graf-after--h4">In clustering problems we split the training examples by unknown characteristics. The algorithm itself decides what characteristic to use for splitting.</p><p name="ea90" id="ea90" class="graf graf--p graf-after--p"><em class="markup--em markup--p-em">Usage examples: market segmentation, social networks analysis, organize computing clusters, astronomical data analysis, image compression, etc.</em></p><ul class="postList"><li name="4a0a" id="4a0a" class="graf graf--li graf-after--p">📗 <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/k_means" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/k_means" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">K-means Algorithm Math</a>&#8202;—&#8202;theory and links for further readings</li><li name="ac60" id="ac60" class="graf graf--li graf-after--li">⚙️ <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/k_means/k_means.py" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/k_means/k_means.py" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">K-means Algorithm Implementation Example</a></li><li name="a212" id="a212" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/k_means/k_means_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/k_means/k_means_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | K-means Algorithm</a>&#8202;—&#8202;split Iris flowers into clusters based on <code class="markup--code markup--li-code">petal_length</code> and <code class="markup--code markup--li-code">petal_width</code></li></ul><h4 name="f32e" id="f32e" class="graf graf--h4 graf-after--li">Neural Networks: Multilayer Perceptron (MLP)</h4><p name="2816" id="2816" class="graf graf--p graf-after--h4">The neural network itself isn’t an algorithm, but rather a framework for many different machine learning algorithms to work together and process complex data inputs.</p><p name="44b5" id="44b5" class="graf graf--p graf-after--p"><em class="markup--em markup--p-em">Usage examples: as a substitute of all other algorithms in general, image recognition, voice recognition, image processing (applying specific style), language translation, etc.</em></p><ul class="postList"><li name="fb52" id="fb52" class="graf graf--li graf-after--p">📗 <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/neural_network" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/neural_network" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Multilayer Perceptron Math</a>&#8202;—&#8202;theory and links for further readings</li><li name="3993" id="3993" class="graf graf--li graf-after--li">⚙️ <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/neural_network/multilayer_perceptron.py" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/neural_network/multilayer_perceptron.py" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Multilayer Perceptron Implementation Example</a></li><li name="7231" id="7231" class="graf graf--li graf-after--li">▶️ <a href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/neural_network/multilayer_perceptron_demo.ipynb" data-href="https://nbviewer.jupyter.org/github/trekhleb/homemade-machine-learning/blob/master/notebooks/neural_network/multilayer_perceptron_demo.ipynb" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">Demo | Multilayer Perceptron</a>&#8202;—&#8202;recognize handwritten digits from <code class="markup--code markup--li-code">28x28</code> pixel images.</li></ul><h4 name="d3b0" id="d3b0" class="graf graf--h4 graf-after--li">Anomaly Detection: Gaussian Distribution</h4><p name="ea70" id="ea70" class="graf graf--p graf-after--h4">Anomaly detection (also outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data.</p><p name="796f" id="796f" class="graf graf--p graf-after--p"><em class="markup--em markup--p-em">Usage examples: intrusion detection, fraud detection, system health monitoring, removing anomalous data from the dataset etc.</em></p><ul class="postList"><li name="5486" id="5486" class="graf graf--li graf-after--p">📗 <a href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/anomaly_detection" data-href="https://github.com/trekhleb/homemade-machine-learning/blob/master/homemade/anomaly_detection" class="markup--anchor markup--li-anchor" rel="nofollow noopener" target="_blank">The Math Behind Anomaly Detection using Gaussian Distribution</a></li></ul><p name="7241" id="7241" class="graf graf--p graf-after--li">—&#8202;—&#8202;—&#8202;—&#8202;—&#8202;—&#8202;—&#8202;—&#8202;—</p><p name="e0dc" id="e0dc" class="graf graf--p graf-after--p graf--trailing">I hope you’ll find <a href="https://github.com/trekhleb/homemade-machine-learning" data-href="https://github.com/trekhleb/homemade-machine-learning" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">the repository</a> useful. Either by playing with demos or by reading the math sections or by simply exploring the source code. Happy coding!</p></div>