diff --git "a/midterm/midterm/take_at_home_(1).ipynb" "b/midterm/midterm/take_at_home_(1).ipynb"
deleted file mode 100644--- "a/midterm/midterm/take_at_home_(1).ipynb"
+++ /dev/null
@@ -1,1335 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "oZ6_2B0E1DAh"
- },
- "source": [
- "# Midterm - Spring 2023\n",
- "\n",
- "## Problem 1: Take-at-home (45 points total)\n",
- "\n",
- "You are applying for a position at the data science team of USDA and you are given data associated with determining appropriate parasite treatment of canines. The suggested treatment options are determined based on a **logistic regression** model that predicts if the canine is infected with a parasite. \n",
- "\n",
- "The data is given in the site: https://data.world/ehales/grls-parasite-study/workspace/file?filename=CBC_data.csv and more specifically in the CBC_data.csv file. Login using you University Google account to access the data and the description that includes a paper on the study (**you dont need to read the paper to solve this problem**). Your target variable $y$ column is titled `parasite_status`. \n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Aq8bln4u1DAo"
- },
- "source": [
- "### Question 1 - Feature Engineering (5 points)\n",
- "\n",
- "Write the posterior probability expressions for logistic regression for the problem you are given to solve."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "_kd85pkA1DA3"
- },
- "source": [
- "$$p(y=1| \\mathbf{x}, \\mathbf w)= \\frac{p(\\mathbf{x}| y=1)p(y=1)}{p(\\mathbf{x}|y=1)p(y=1)+p(\\mathbf{x}|y=0)}=\\frac{1}{1+\\exp(-\\alpha)}=\\sigma(\\alpha)$$\n",
- "\n",
- "$$p(y=0| \\mathbf{x}, \\mathbf w)=1-p(y=1|\\mathbf{x}^{T}\\mathbf{w})=1-\\sigma(\\alpha)=\\sigma(-\\alpha)$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Gh6Fi5hz1DA6"
- },
- "source": [
- "\n",
- "\n",
- "### Question 2 - Decision Boundary (5 points)\n",
- "\n",
- "Write the expression for the decision boundary assuming that $p(y=1)=p(y=0)$. The decision boundary is the line that separates the two classes.\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "HMr2tF_J1DA-"
- },
- "source": [
- "$$p(y=1)=p(y=0)→\\sigma(\\alpha)=-\\sigma(\\alpha)→2\\sigma(\\alpha)=1→\\sigma(\\alpha)=0.5≡\\sigma(\\mathbf{w}^T\\mathbf{x})=0.5$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "750Hn0iC1DBA"
- },
- "source": [
- "\n",
- "\n",
- "### Question 3 - Loss function (5 points)\n",
- "\n",
- "Write the expression of the loss as a function of $\\mathbf w$ that makes sense for you to use in this problem. \n",
- "\n",
- "NOTE: The loss will be a function that will include this function: \n",
- "\n",
- "$$\\sigma(a) = \\frac{1}{1+e^{-a}}$$\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jxiR0jEh1DBD"
- },
- "source": [
- "$$\n",
- "\\begin{align}\n",
- "L_{CE} = -[\\sum_{i=1}^m \\{y_i\\ln \\hat{y}_i + (1-y_i)\\ln(1-\\hat{y}_i)\\}]\\\\\n",
- "= -[\\sum_{i=1}^m\\{y_i\\ln\\frac{1}{1+\\exp(-\\mathbf{w}^T\\mathbf{x})}+(1-y_i)\\ln(1-\\frac{1}{1+\\exp(-\\mathbf{w}^T\\mathbf{x})})\\}] \\\\\n",
- "= -[\\sum_{i=1}^m\\{y_i[\\ln\\frac{1}{1+\\exp(-\\alpha)}-\\ln(1-\\frac{1}{1+\\exp(-\\alpha)})]+\\ln(1-\\frac{1}{1+\\exp(-\\alpha)})\\}] \\\\\n",
- "= -[\\sum_{i=1}^m\\{y_i\\mathbf{w}^T\\mathbf{x}-\\ln(1+\\exp(\\mathbf{w}^T\\mathbf{x}))\\}]\n",
- "\\end{align}\n",
- "$$\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AW4xA4221DBF"
- },
- "source": [
- "\n",
- "### Question 4 - Gradient (5 points)\n",
- "\n",
- "Write the expression of the gradient of the loss with respect to the parameters - show all your work.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "bo0YDA0i1DBJ"
- },
- "source": [
- "$$\n",
- "\\begin{align}\n",
- "\\nabla_\\mathbf w L_{CE} = \\nabla_\\mathbf{w}-[\\sum_{i=1}^m\\{y_i\\mathbf{w}^T\\mathbf{x}-ln(1+\\exp(\\mathbf{w}^T\\mathbf{x})\\}] \\\\\n",
- "= [-\\sum_{i=1}^my_ix_i] + [\\sum_{i=1}^m\\frac{1}{1+\\exp(\\mathbf{w}^T\\mathbf{x})}\\exp(\\mathbf{w}^T\\mathbf{x})*x_i] \\\\\n",
- "= [-\\sum_{i=1}^my_ix_i] + [\\sum_{i=1}^m(\\sigma(\\mathbf{w}^T\\mathbf{x}))*x_i] \\\\\n",
- "= \\sum_{i=1}^m (\\sigma(\\mathbf{w}^T\\mathbf{x})-y_i)x_i= \\sum_{i=1}^m(\\hat{y}_i-y_i)x_i\n",
- "\\end{align}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "BpUryvTT1DBM"
- },
- "source": [
- "### Question 5 - Imbalanced dataset (10 points)\n",
- "\n",
- "You are now told that in the dataset \n",
- "\n",
- "$$p(y=0) >> p(y=1)$$\n",
- "\n",
- "Can you comment if the accuracy of Logistic Regression will be affected by such imbalance?\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "TqdImYQf1DBP"
- },
- "source": [
- "We know that the loss function heavily penalizes confident wrong decisions. We expect then, that the model will be strongly incentivized to predict 0 more frequently than 1, regardless of the true outcome, as this minimizes loss. This will cause more false negatives, and will need to be considered with regards to our ROC curve. The accuracy will be affected, as there are so few positive examples that the model cannot accurately learn them."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "FK1Su76R1DBS"
- },
- "source": [
- "\n",
- "### Question 6 - SGD (15 points)\n",
- "\n",
- "The interviewer was impressed with your answers and wants to test your programming skills. \n",
- "\n",
- "1. Use the dataset to train a logistic regressor that will predict the target variable $y$. \n",
- "\n",
- " 2. Report the harmonic mean of precision (p) and recall (r) i.e the [metric called $F_1$ score](https://en.wikipedia.org/wiki/F-score) that is calculated as shown below using a test dataset that is 20% of each group. Plot the $F_1$ score vs the iteration number $t$. \n",
- "\n",
- "$$F_1 = \\frac{2}{r^{-1} + p^{-1}}$$\n",
- "\n",
- "Your code includes hyperparameter optimization of the learning rate and mini batch size. Please learn about cross validation which is a splitting strategy for tuning models [here](https://scikit-learn.org/stable/modules/cross_validation.html).\n",
- "\n",
- "You are allowed to use any library you want to code this problem.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "id": "cnxqYSvL1DBV"
- },
- "outputs": [],
- "source": [
- "# write your code here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Preprocessing"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " ID \n",
- " SEX \n",
- " TYPEAREA \n",
- " SEX.REPRO \n",
- " REPRO.STATUS \n",
- " AGE \n",
- " PARASITE_STATUS \n",
- " RBC \n",
- " HGB \n",
- " WBC \n",
- " EOS.CNT \n",
- " MONO.CNT \n",
- " NUT.CNT \n",
- " PL.CNT \n",
- " LYMP.CNT \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " grls5ZUT2BYY \n",
- " Male \n",
- " Suburban \n",
- " IntactMale \n",
- " Intact \n",
- " 9 \n",
- " Negative \n",
- " 6.4 \n",
- " 16.6 \n",
- " 14.2 \n",
- " 142.0 \n",
- " 852.0 \n",
- " 6390.0 \n",
- " 210.0 \n",
- " 6816.0 \n",
- " \n",
- " \n",
- " 1 \n",
- " grls8DCONYUU \n",
- " Female \n",
- " Rural \n",
- " NeuteredFemale \n",
- " Neutered \n",
- " 6 \n",
- " Negative \n",
- " 4.8 \n",
- " 12.5 \n",
- " 10.0 \n",
- " 400.0 \n",
- " 300.0 \n",
- " 4800.0 \n",
- " 209.0 \n",
- " 4500.0 \n",
- " \n",
- " \n",
- " 2 \n",
- " grlsUC5R4PTT \n",
- " Male \n",
- " Suburban \n",
- " IntactMale \n",
- " Intact \n",
- " 14 \n",
- " Negative \n",
- " 6.2 \n",
- " 17.3 \n",
- " 9.5 \n",
- " 190.0 \n",
- " 475.0 \n",
- " 7315.0 \n",
- " 164.0 \n",
- " 1520.0 \n",
- " \n",
- " \n",
- " 3 \n",
- " grlsXUR2PY88 \n",
- " Male \n",
- " Rural \n",
- " IntactMale \n",
- " Intact \n",
- " 6 \n",
- " Negative \n",
- " 5.4 \n",
- " 13.8 \n",
- " 14.1 \n",
- " 1692.0 \n",
- " 423.0 \n",
- " 7755.0 \n",
- " 254.0 \n",
- " 4230.0 \n",
- " \n",
- " \n",
- " 4 \n",
- " grlsTBZUF3GG \n",
- " Female \n",
- " Rural \n",
- " IntactFemale \n",
- " Intact \n",
- " 18 \n",
- " Negative \n",
- " 5.9 \n",
- " 14.4 \n",
- " 6.5 \n",
- " 390.0 \n",
- " 130.0 \n",
- " 2795.0 \n",
- " 213.0 \n",
- " 3185.0 \n",
- " \n",
- " \n",
- "
\n",
- "
\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " SEX \n",
- " TYPEAREA \n",
- " SEX.REPRO \n",
- " REPRO.STATUS \n",
- " AGE \n",
- " PARASITE_STATUS \n",
- " RBC \n",
- " HGB \n",
- " WBC \n",
- " EOS.CNT \n",
- " MONO.CNT \n",
- " NUT.CNT \n",
- " PL.CNT \n",
- " LYMP.CNT \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 1 \n",
- " Suburban \n",
- " IntactMale \n",
- " 0 \n",
- " 9 \n",
- " 0 \n",
- " 6.4 \n",
- " 16.6 \n",
- " 14.2 \n",
- " 142.0 \n",
- " 852.0 \n",
- " 6390.0 \n",
- " 210.0 \n",
- " 6816.0 \n",
- " \n",
- " \n",
- " 1 \n",
- " 0 \n",
- " Rural \n",
- " NeuteredFemale \n",
- " 1 \n",
- " 6 \n",
- " 0 \n",
- " 4.8 \n",
- " 12.5 \n",
- " 10.0 \n",
- " 400.0 \n",
- " 300.0 \n",
- " 4800.0 \n",
- " 209.0 \n",
- " 4500.0 \n",
- " \n",
- " \n",
- " 2 \n",
- " 1 \n",
- " Suburban \n",
- " IntactMale \n",
- " 0 \n",
- " 14 \n",
- " 0 \n",
- " 6.2 \n",
- " 17.3 \n",
- " 9.5 \n",
- " 190.0 \n",
- " 475.0 \n",
- " 7315.0 \n",
- " 164.0 \n",
- " 1520.0 \n",
- " \n",
- " \n",
- " 3 \n",
- " 1 \n",
- " Rural \n",
- " IntactMale \n",
- " 0 \n",
- " 6 \n",
- " 0 \n",
- " 5.4 \n",
- " 13.8 \n",
- " 14.1 \n",
- " 1692.0 \n",
- " 423.0 \n",
- " 7755.0 \n",
- " 254.0 \n",
- " 4230.0 \n",
- " \n",
- " \n",
- " 4 \n",
- " 0 \n",
- " Rural \n",
- " IntactFemale \n",
- " 0 \n",
- " 18 \n",
- " 0 \n",
- " 5.9 \n",
- " 14.4 \n",
- " 6.5 \n",
- " 390.0 \n",
- " 130.0 \n",
- " 2795.0 \n",
- " 213.0 \n",
- " 3185.0 \n",
- " \n",
- " \n",
- "
\n",
- "
\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " SEX \n",
- " REPRO.STATUS \n",
- " AGE \n",
- " PARASITE_STATUS \n",
- " TYPEAREA_Rural \n",
- " TYPEAREA_Suburban \n",
- " TYPEAREA_Urban \n",
- " SEX.REPRO_IntactFemale \n",
- " SEX.REPRO_IntactMale \n",
- " SEX.REPRO_NeuteredFemale \n",
- " SEX.REPRO_NeuteredMale \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 9 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " \n",
- " \n",
- " 1 \n",
- " 0 \n",
- " 1 \n",
- " 6 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " \n",
- " \n",
- " 2 \n",
- " 1 \n",
- " 0 \n",
- " 14 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " \n",
- " \n",
- " 3 \n",
- " 1 \n",
- " 0 \n",
- " 6 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " \n",
- " \n",
- " 4 \n",
- " 0 \n",
- " 0 \n",
- " 18 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 1 \n",
- " 0 \n",
- " 0 \n",
- " 0 \n",
- " \n",
- " \n",
- "
\n",
- "
\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " epoch \n",
- " f1 score \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 0 \n",
- " 0.830236 \n",
- " \n",
- " \n",
- " 1 \n",
- " 1 \n",
- " 0.846667 \n",
- " \n",
- " \n",
- " 2 \n",
- " 2 \n",
- " 0.846844 \n",
- " \n",
- " \n",
- " 3 \n",
- " 3 \n",
- " 0.846762 \n",
- " \n",
- " \n",
- " 4 \n",
- " 4 \n",
- " 0.850135 \n",
- " \n",
- " \n",
- "
\n",
- "
"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "import seaborn as sns\n",
- "import matplotlib.pyplot as plt\n",
- "\n",
- "def plot_f1(f1s: pd.DataFrame):\n",
- " sns.scatterplot(x=f1s['epoch'], y=f1s['f1 score'])\n",
- " sns.lineplot(x=f1s['epoch'], y=f1s['f1 score'])\n",
- " plt.show()\n",
- "plot_f1(f1s)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Hyperparameter optimization"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 41,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "from sklearn.pipeline import Pipeline #make_pipeline\n",
- "from sklearn import preprocessing\n",
- "from imblearn.over_sampling import RandomOverSampler\n",
- "#from imblearn.pipeline import Pipeline\n",
- "#from sklearn.linear_model import LogisticRegression, SGDClassifier\n",
- "\n",
- "#i = 20000\n",
- "\n",
- "pipe = Pipeline([\n",
- "# ('oversampler', RandomOverSampler()),\n",
- " ('preprocessor', preprocessing.StandardScaler()),\n",
- "# ('classifier', LogisticRegression(max_iter=i, class_weight='balanced')),\n",
- " ('classifier', MyLogisticRegression())\n",
- "])\n",
- "\n",
- "\n",
- "\n",
- "#model = LogisticRegression(penalty='l1', solver='saga', max_iter=i)\n",
- "#pipe.fit(X_train, y_train)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 42,
- "metadata": {
- "scrolled": true,
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "{'classifier__batch_size': 82, 'classifier__lr': 0.1}"
- ]
- },
- "execution_count": 42,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from sklearn.model_selection import GridSearchCV\n",
- "import numpy as np\n",
- "\n",
- "# optimize hyperparameter batch size and learning rate\n",
- "param_grid = {\n",
- " #'classifier__penalty': ['l1', 'l2'],\n",
- " #'classifier__C': [1e-4, 1e-3, 1e-2, 0.1, 1, 10],\n",
- " #'classifier__solver': ['sag'],\n",
- " #'classifier__class_weight': [None, 'balanced', *[{0: 1, 1:10**x} for x in range(-5, 5)]],\n",
- " #'classifier__eta0': [10**x for x in range(-5, 5)],\n",
- " 'classifier__batch_size': np.linspace(1, X_train.shape[0], 70, dtype=int)[:10],\n",
- " 'classifier__lr': [10**x for x in range(-5, 5)],\n",
- "}\n",
- "\n",
- "grid_search = GridSearchCV(pipe, param_grid, cv=5, scoring='f1', n_jobs=-1)\n",
- "#grid_search = GridSearchCV(MyLogisticRegression(), param_grid, cv=5, scoring='f1', n_jobs=-1)\n",
- "\n",
- "grid_search.fit(X_train, y_train)\n",
- "grid_search.best_params_"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 43,
- "metadata": {
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0.9166666666666666"
- ]
- },
- "execution_count": 43,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from sklearn.metrics import precision_score, recall_score, f1_score\n",
- "from sklearn.metrics import confusion_matrix\n",
- "\n",
- "y_pred = grid_search.predict(X_test)\n",
- "precision = precision_score(y_test, y_pred)\n",
- "precision"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 44,
- "metadata": {
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "array([[ 33, 12],\n",
- " [123, 132]])"
- ]
- },
- "execution_count": 44,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "confusion_matrix(y_pred == y_test, y_pred)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 45,
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "#from sklearn.model_selection import cross_val_score\n",
- "#\n",
- "#scores = cross_val_score(grid_search, X, y, cv=5)\n",
- "#scores"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 46,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- "\n",
- "
\n",
- " \n",
- " \n",
- " epoch \n",
- " f1 score \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " 0 \n",
- " 0 \n",
- " 0.813799 \n",
- " \n",
- " \n",
- " 1 \n",
- " 1 \n",
- " 0.823768 \n",
- " \n",
- " \n",
- " 2 \n",
- " 2 \n",
- " 0.833749 \n",
- " \n",
- " \n",
- " 3 \n",
- " 3 \n",
- " 0.837076 \n",
- " \n",
- " \n",
- " 4 \n",
- " 4 \n",
- " 0.837068 \n",
- " \n",
- " \n",
- "
\n",
- "
"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "plot_f1(best_f1)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- },
- "vscode": {
- "interpreter": {
- "hash": "62556f7a043365a66e0918c892755cfafede529a87e97207556f006a109bade4"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}