mypytorch 0.0.5
My Pytorch bindings
License: MIT
Keywords: python,pytorch

# MyPytorch

`MyPytorch` provides convenient functionality to abstract some of `Pytorch` tedious and repetitive work but without compromising on flexibility.


```python
from mypytorch import *
```

Let's see how to work with `MyPytorch` with the following example. First, we download some data (in this case we work with MNIST).


```python
# get some data

from sklearn.datasets import fetch_openml
import numpy as np

X, y = fetch_openml(
    'mnist_784', 
    version=1, 
    return_X_y=True)

y = y.astype(np.int)
```


```python
# visualize images

import matplotlib.pyplot as plt
import random

fig, axs = plt.subplots(3,5, figsize=(10,7))
for _ax in axs:
    for ax in _ax:
        ix = random.randint(0, len(X)-1)
        img = X[ix].reshape((28,28))
        ax.imshow(img)
        ax.set_title(y[ix])
        ax.axis('off')

plt.show()
```


![png](pics/output_6_0.png)



```python
# train-validation-test split

X_train, y_train, X_test, y_test = X[:1100] / 255., y[:1100], X[1100:1200] / 255., y[1100:1200]
X_train, y_train, X_eval, y_eval = X_train[:1000], y_train[:1000], X_train[1000:], y_train[1000:]

X_train.shape, y_train.shape, X_eval.shape, y_eval.shape, X_test.shape, y_test.shape
```




    ((1000, 784), (1000,), (100, 784), (100,), (100, 784), (100,))



The basic module of `MyPytorch` is the `Model`, and in order to define a model we first need a `Network`. You define a neural network like in `Pytorch`.


```python
# define a network

import torch

class Net(torch.nn.Module):
    
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(28*28, 100)
        self.relu = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(100, 10)
    
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x
    
net = Net()
net(torch.randn(32, 28*28)).shape
```




    torch.Size([32, 10])



The `Model` is instantiated with the `Network` but in order to train it we need to `compile` it with an `optimizer` and a `loss` function. We can define them just like in `Pytorch`.


```python
# define a model

net = Net()
model = MyModel(net)

# define optimizer and loss function

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()

# compile the model

model.compile(optimizer, loss)
```

To train the `Network` we call the `fit` function of our `Model`. We only need to pass a `Dataloader`. You can define your own dataloader and it will work as long as you can iterate on it.


```python
# train the network

batch_size = 32
batches = len(X_train) // batch_size
dataloader = [
    (torch.tensor(X_train[i*batch_size:(i+1)*batch_size]).float(), 
     torch.tensor(y_train[i*batch_size:(i+1)*batch_size]).long()) 
    for i in range(batches)
]

hist = model.fit(dataloader)
```


Epoch 1/10  loss 1.99136<p>Epoch 2/10  loss 1.15712<p>Epoch 3/10  loss 0.71965<p>Epoch 4/10  loss 0.54304<p>Epoch 5/10  loss 0.44808<p>Epoch 6/10  loss 0.38635<p>Epoch 7/10  loss 0.34145<p>Epoch 8/10  loss 0.30618<p>Epoch 9/10  loss 0.27706<p>Epoch 10/10  loss 0.25207


You can (and should) use `Pytorch` own `DataLoaders`.


```python
from torch.utils.data import Dataset, DataLoader

class MyDataset(Dataset):
    def __init__(self, X, y):
        self.X = X
        self.y = y
    def __len__(self):
        return len(self.X)
    def __getitem__(self, ix):
        return torch.tensor(self.X[ix]).float(), torch.tensor(self.y[ix]).long()
    
dataset = MyDataset(X_train, y_train)
datalaoder = DataLoader(dataset, batch_size=batch_size, shuffle=True)
```


```python
net = Net()
model = MyModel(net)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
model.compile(optimizer, loss)
hist = model.fit(dataloader)
```


Epoch 1/10  loss 1.95815<p>Epoch 2/10  loss 1.10758<p>Epoch 3/10  loss 0.69818<p>Epoch 4/10  loss 0.53266<p>Epoch 5/10  loss 0.44248<p>Epoch 6/10  loss 0.38358<p>Epoch 7/10  loss 0.3405<p>Epoch 8/10  loss 0.30634<p>Epoch 9/10  loss 0.27786<p>Epoch 10/10  loss 0.25341


You can pass a list of `Metrics` to the `compile` function in order to evaluate the `Network` during training. 


```python
# adding metrics

net = Net()
model = MyModel(net)

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy()]

model.compile(optimizer, loss, metrics)

hist = model.fit(dataloader)
```


Epoch 1/10  loss 1.99799 acc 0.4869<p>Epoch 2/10  loss 1.15223 acc 0.76915<p>Epoch 3/10  loss 0.6995 acc 0.8377<p>Epoch 4/10  loss 0.5272 acc 0.86694<p>Epoch 5/10  loss 0.43763 acc 0.88407<p>Epoch 6/10  loss 0.38001 acc 0.89214<p>Epoch 7/10  loss 0.33786 acc 0.90625<p>Epoch 8/10  loss 0.30444 acc 0.91734<p>Epoch 9/10  loss 0.27655 acc 0.92843<p>Epoch 10/10  loss 0.25238 acc 0.93246


A `Metric` is just a regular class with a `name` and the corresponding function to compute the metric. You can use multiple `Metrics` and also write your own.


```python
from sklearn.metrics import f1_score

class F1():
    def __init__(self):
        self.name = "F1"
    def __call__(self, y_hat, y):
        return f1_score(y.cpu(), torch.argmax(y_hat, axis=1).cpu().detach(), average='macro').item()

net = Net()
model = MyModel(net)

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

hist = model.fit(dataloader)
```


Epoch 1/10  loss 1.97591 acc 0.51008 F1 0.4544<p>Epoch 2/10  loss 1.11875 acc 0.75605 F1 0.71131<p>Epoch 3/10  loss 0.70105 acc 0.83367 F1 0.80737<p>Epoch 4/10  loss 0.53526 acc 0.86694 F1 0.84931<p>Epoch 5/10  loss 0.44538 acc 0.87601 F1 0.86108<p>Epoch 6/10  loss 0.38682 acc 0.89415 F1 0.87824<p>Epoch 7/10  loss 0.34412 acc 0.90524 F1 0.8888<p>Epoch 8/10  loss 0.31014 acc 0.91532 F1 0.90095<p>Epoch 9/10  loss 0.28198 acc 0.92339 F1 0.91297<p>Epoch 10/10  loss 0.25767 acc 0.92944 F1 0.92055


You can pass data for evaluation during training as a second parameter of the `compile` function. Evaluation data must be also a `Dataloader`.


```python
# adding evaluation data

eval_dataset = MyDataset(X_eval, y_eval)
eval_dataloader = DataLoader(eval_dataset, batch_size=batch_size, shuffle=True)
        
net = Net()
model = MyModel(net)

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

hist = model.fit(dataloader, eval_dataloader)
```


Epoch 1/10  loss 1.97824 acc 0.54435 F1 0.476  eval_loss 1.62018 eval_acc 0.70312 eval_F1 0.52057<p>Epoch 2/10  loss 1.12812 acc 0.7621 F1 0.71212  eval_loss 1.03902 eval_acc 0.75 eval_F1 0.62577<p>Epoch 3/10  loss 0.70452 acc 0.83972 F1 0.81291  eval_loss 0.79963 eval_acc 0.78906 eval_F1 0.6761<p>Epoch 4/10  loss 0.53748 acc 0.8629 F1 0.84499  eval_loss 0.89987 eval_acc 0.74219 eval_F1 0.66277<p>Epoch 5/10  loss 0.44723 acc 0.87802 F1 0.86192  eval_loss 0.65931 eval_acc 0.79688 eval_F1 0.74391<p>Epoch 6/10  loss 0.38843 acc 0.89113 F1 0.87569  eval_loss 0.67273 eval_acc 0.79688 eval_F1 0.68275<p>Epoch 7/10  loss 0.3454 acc 0.90625 F1 0.89326  eval_loss 0.57253 eval_acc 0.84375 eval_F1 0.8178<p>Epoch 8/10  loss 0.31145 acc 0.91935 F1 0.9052  eval_loss 0.71521 eval_acc 0.78906 eval_F1 0.71851<p>Epoch 9/10  loss 0.28318 acc 0.92843 F1 0.91801  eval_loss 0.60076 eval_acc 0.85156 eval_F1 0.81746<p>Epoch 10/10  loss 0.25873 acc 0.93145 F1 0.92216  eval_loss 0.55954 eval_acc 0.85156 eval_F1 0.76582


By default computations are executed on a `GPU` if available (if not the model will run in the `CPU`). You can specify your device in different ways.


```python
net = Net()
model = MyModel(net, device="cpu") # now the default device is CPU for everything

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

hist = model.fit(dataloader, eval_dataloader, epochs=2)
```


Epoch 1/2  loss 1.96979 acc 0.53024 F1 0.46063  eval_loss 1.47816 eval_acc 0.71094 eval_F1 0.62278<p>Epoch 2/2  loss 1.1105 acc 0.76613 F1 0.72816  eval_loss 1.18418 eval_acc 0.70312 eval_F1 0.60498



```python
net = Net()
model = MyModel(net) 

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

# fit on CPU, but the default device still is GPU (if available)

hist = model.fit(dataloader, eval_dataloader, epochs=10, device="cpu") 
```


Epoch 1/10  loss 1.9707 acc 0.50403 F1 0.43994  eval_loss 1.51756 eval_acc 0.67969 eval_F1 0.5174<p>Epoch 2/10  loss 1.12484 acc 0.7631 F1 0.72922  eval_loss 1.01289 eval_acc 0.79688 eval_F1 0.75185<p>Epoch 3/10  loss 0.69954 acc 0.8377 F1 0.81406  eval_loss 1.00583 eval_acc 0.71094 eval_F1 0.6564<p>Epoch 4/10  loss 0.53259 acc 0.86895 F1 0.85148  eval_loss 0.72211 eval_acc 0.78906 eval_F1 0.71051<p>Epoch 5/10  loss 0.44368 acc 0.88306 F1 0.86737  eval_loss 0.84888 eval_acc 0.74219 eval_F1 0.66741<p>Epoch 6/10  loss 0.38588 acc 0.89113 F1 0.87572  eval_loss 0.60628 eval_acc 0.85938 eval_F1 0.80177<p>Epoch 7/10  loss 0.34355 acc 0.90121 F1 0.88637  eval_loss 0.67603 eval_acc 0.78906 eval_F1 0.69369<p>Epoch 8/10  loss 0.30982 acc 0.91734 F1 0.90263  eval_loss 0.57527 eval_acc 0.85156 eval_F1 0.80583<p>Epoch 9/10  loss 0.28156 acc 0.9244 F1 0.91464  eval_loss 0.61433 eval_acc 0.8125 eval_F1 0.755<p>Epoch 10/10  loss 0.25711 acc 0.93347 F1 0.92495  eval_loss 0.87156 eval_acc 0.82031 eval_F1 0.75841


The `fit` function will return the `history` of the training so you can plot training curves.


```python
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(15,5))
plt.subplot(121)
plt.plot(hist["epochs"], hist["metrics"]["loss"], label="loss")
plt.plot(hist["epochs"], hist["metrics"]["eval_loss"], label="eval loss")
plt.grid(True)
plt.xlabel("Epoch")
plt.legend()
plt.subplot(122)
plt.plot(hist["epochs"], hist["metrics"]["acc"], label="acc")
plt.plot(hist["epochs"], hist["metrics"]["eval_acc"], label="eval acc")
plt.plot(hist["epochs"], hist["metrics"]["F1"], label="F1")
plt.plot(hist["epochs"], hist["metrics"]["eval_F1"], label="eval F1")
plt.xlabel("Epoch")
plt.legend()
plt.grid(True)
plt.show()
```


![png](pics/output_27_0.png)


You can `evaluate` your model to get some metrics like this 


```python
model.evaluate(eval_dataloader)
```


eval_loss 0.52886 eval_acc 0.875 eval_F1 0.84095


And get some predictions with the `predict` method


```python
test_dataloader = [torch.tensor(X_test[:10]).float()]

y_hat = model.predict(test_dataloader)
probas = torch.softmax(y_hat, axis = 1)
preds = torch.argmax(probas, axis = 1)

preds, y_test[:10]
```



<div>
    <style>
        /* Turns off some styling */
        progress {
            /* gets rid of default border in Firefox and Opera. */
            border: none;
            /* Needs to be in here for Safari polyfill so background images work as expected. */
            background-size: auto;
        }
        .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
            background: #F44336;
        }
    </style>
  <progress value='1' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [1/1 00:00<00:00]
</div>






    (tensor([6, 8, 5, 9, 5, 8, 7, 0, 3, 5], device='cuda:0',
            grad_fn=<NotImplemented>),
     array([6, 8, 0, 9, 5, 8, 7, 0, 3, 5]))



## Flexibility

You can use our building blocks to do more complicated stuff like `early stopping` or `learning rate scheduling` (although we support that).


```python
net = Net()
model = MyModel(net) 

optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

epochs = 100
best_loss, step, early_stop = 1e10, 0, 3
for e in range(1, epochs+1):
    
    train_metrics = model.train(dataloader)    
    eval_metrics = model.eval(eval_dataloader, device="cpu")
    print(f"Epoch {e}/{epochs} {train_metrics} {eval_metrics}")
    
    # early stop
    eval_loss = eval_metrics['eval_loss']
    step = step + 1
    if eval_loss < best_loss:
        best_loss = eval_loss
        step = 0
    if step >= early_stop:
        print(f"Early stopping at epoch {e}")
        break
```

    Epoch 1/100  loss 2.28437 acc 0.14214 F1 0.09496  eval_loss 2.24477 eval_acc 0.15625 eval_F1 0.10752
    Epoch 2/100  loss 2.22458 acc 0.30948 F1 0.26141  eval_loss 2.1856 eval_acc 0.5 eval_F1 0.40614
    Epoch 3/100  loss 2.15845 acc 0.50101 F1 0.44378  eval_loss 2.13385 eval_acc 0.57031 eval_F1 0.43026
    Epoch 4/100  loss 2.08266 acc 0.60484 F1 0.54757  eval_loss 2.05999 eval_acc 0.5625 eval_F1 0.46542
    Epoch 5/100  loss 1.99796 acc 0.6623 F1 0.60319  eval_loss 1.9568 eval_acc 0.71094 eval_F1 0.56703
    Epoch 6/100  loss 1.90427 acc 0.69153 F1 0.63079  eval_loss 1.8793 eval_acc 0.76562 eval_F1 0.70411
    Epoch 7/100  loss 1.80246 acc 0.70968 F1 0.65215  eval_loss 1.79829 eval_acc 0.75781 eval_F1 0.67091
    Epoch 8/100  loss 1.69496 acc 0.72581 F1 0.67075  eval_loss 1.75112 eval_acc 0.77344 eval_F1 0.68644
    Epoch 9/100  loss 1.58603 acc 0.73589 F1 0.67894  eval_loss 1.57126 eval_acc 0.73438 eval_F1 0.65294
    Epoch 10/100  loss 1.47924 acc 0.74294 F1 0.69032  eval_loss 1.58471 eval_acc 0.6875 eval_F1 0.57087
    Epoch 11/100  loss 1.37742 acc 0.75504 F1 0.70785  eval_loss 1.50139 eval_acc 0.75 eval_F1 0.63293
    Epoch 12/100  loss 1.28234 acc 0.77016 F1 0.7249  eval_loss 1.34448 eval_acc 0.82812 eval_F1 0.78028
    Epoch 13/100  loss 1.19535 acc 0.77823 F1 0.73671  eval_loss 1.32964 eval_acc 0.71875 eval_F1 0.63572
    Epoch 14/100  loss 1.11691 acc 0.78327 F1 0.7467  eval_loss 1.1703 eval_acc 0.83594 eval_F1 0.79638
    Epoch 15/100  loss 1.04675 acc 0.79839 F1 0.76401  eval_loss 1.28964 eval_acc 0.73438 eval_F1 0.66533
    Epoch 16/100  loss 0.98435 acc 0.81149 F1 0.78336  eval_loss 1.19126 eval_acc 0.78125 eval_F1 0.6906
    Epoch 17/100  loss 0.9289 acc 0.82157 F1 0.7928  eval_loss 1.07859 eval_acc 0.83594 eval_F1 0.77242
    Epoch 18/100  loss 0.87959 acc 0.82863 F1 0.80025  eval_loss 1.04709 eval_acc 0.78125 eval_F1 0.62931
    Epoch 19/100  loss 0.83565 acc 0.83569 F1 0.81038  eval_loss 0.95057 eval_acc 0.83594 eval_F1 0.73592
    Epoch 20/100  loss 0.79636 acc 0.84173 F1 0.8185  eval_loss 0.93946 eval_acc 0.83594 eval_F1 0.76038
    Epoch 21/100  loss 0.76113 acc 0.84677 F1 0.82489  eval_loss 1.02549 eval_acc 0.78125 eval_F1 0.70139
    Epoch 22/100  loss 0.72943 acc 0.8498 F1 0.82764  eval_loss 0.91239 eval_acc 0.83594 eval_F1 0.80488
    Epoch 23/100  loss 0.70078 acc 0.85181 F1 0.83018  eval_loss 0.94465 eval_acc 0.77344 eval_F1 0.63058
    Epoch 24/100  loss 0.67479 acc 0.85887 F1 0.83703  eval_loss 0.9098 eval_acc 0.77344 eval_F1 0.6976
    Epoch 25/100  loss 0.6511 acc 0.86089 F1 0.84087  eval_loss 0.79746 eval_acc 0.82812 eval_F1 0.77353
    Epoch 26/100  loss 0.62944 acc 0.86492 F1 0.84596  eval_loss 0.93608 eval_acc 0.78125 eval_F1 0.70442
    Epoch 27/100  loss 0.60954 acc 0.86794 F1 0.84983  eval_loss 0.79635 eval_acc 0.84375 eval_F1 0.75108
    Epoch 28/100  loss 0.5912 acc 0.875 F1 0.85971  eval_loss 0.85732 eval_acc 0.79688 eval_F1 0.71984
    Epoch 29/100  loss 0.57424 acc 0.875 F1 0.85881  eval_loss 0.7743 eval_acc 0.79688 eval_F1 0.70714
    Epoch 30/100  loss 0.55849 acc 0.87601 F1 0.8598  eval_loss 0.80398 eval_acc 0.85156 eval_F1 0.78112
    Epoch 31/100  loss 0.54383 acc 0.88004 F1 0.86276  eval_loss 0.9629 eval_acc 0.74219 eval_F1 0.65322
    Epoch 32/100  loss 0.53014 acc 0.88306 F1 0.86988  eval_loss 0.77224 eval_acc 0.85156 eval_F1 0.80185
    Epoch 33/100  loss 0.51732 acc 0.88407 F1 0.87062  eval_loss 0.8065 eval_acc 0.80469 eval_F1 0.72396
    Epoch 34/100  loss 0.5053 acc 0.88508 F1 0.87196  eval_loss 0.67485 eval_acc 0.85938 eval_F1 0.8332
    Epoch 35/100  loss 0.49399 acc 0.8881 F1 0.87297  eval_loss 1.03528 eval_acc 0.80469 eval_F1 0.71252
    Epoch 36/100  loss 0.48333 acc 0.88911 F1 0.87394  eval_loss 0.79309 eval_acc 0.80469 eval_F1 0.74181
    Epoch 37/100  loss 0.47325 acc 0.89113 F1 0.87593  eval_loss 0.70955 eval_acc 0.86719 eval_F1 0.81513
    Early stopping at epoch 37
    

Or even build your custom `Model` (you will have access to the `optimizer`, `loss` function, `metrics`, batches, etc).


```python
class MyCustomModel(MyModel):
    
    # build your own prediction function
    
    def predict(self, dataset, device="cpu"):
        self.net.to(device)
        self.net.eval()
        preds = []
        for X in dataset:
            X = torch.tensor(X).float()
            X = X.to(device)
            y_hat = self.net(X.unsqueeze(0))
            probas = torch.softmax(y_hat, axis=1)
            pred = torch.argmax(probas, axis=1)
            preds.append(pred.item())
        return preds
```


```python
net = Net()
model = MyCustomModel(net) 

optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
loss = torch.nn.CrossEntropyLoss()
metrics = [Accuracy(), F1()]

model.compile(optimizer, loss, metrics)

hist = model.fit(dataloader, eval_dataloader)
```


Epoch 1/10  loss 1.95079 acc 0.5121 F1 0.444  eval_loss 1.45498 eval_acc 0.72656 eval_F1 0.66349<p>Epoch 2/10  loss 1.10137 acc 0.76815 F1 0.72531  eval_loss 1.08671 eval_acc 0.75781 eval_F1 0.65991<p>Epoch 3/10  loss 0.69632 acc 0.83972 F1 0.81439  eval_loss 1.02583 eval_acc 0.78906 eval_F1 0.74668<p>Epoch 4/10  loss 0.53076 acc 0.86492 F1 0.84749  eval_loss 0.70602 eval_acc 0.85938 eval_F1 0.80634<p>Epoch 5/10  loss 0.44148 acc 0.88004 F1 0.86564  eval_loss 0.6754 eval_acc 0.86719 eval_F1 0.81322<p>Epoch 6/10  loss 0.38359 acc 0.89617 F1 0.88165  eval_loss 0.6672 eval_acc 0.86719 eval_F1 0.8592<p>Epoch 7/10  loss 0.34096 acc 0.90323 F1 0.88823  eval_loss 0.55951 eval_acc 0.86719 eval_F1 0.80959<p>Epoch 8/10  loss 0.307 acc 0.91633 F1 0.90295  eval_loss 0.63003 eval_acc 0.82031 eval_F1 0.67254<p>Epoch 9/10  loss 0.27858 acc 0.92742 F1 0.91722  eval_loss 0.52667 eval_acc 0.86719 eval_F1 0.83345<p>Epoch 10/10  loss 0.25406 acc 0.93145 F1 0.92172  eval_loss 0.52484 eval_acc 0.86719 eval_F1 0.81528



```python
preds = model.predict(X_test)

preds[:10], y_test[:10]
```




    ([6, 8, 5, 9, 5, 8, 7, 0, 3, 5], array([6, 8, 0, 9, 5, 8, 7, 0, 3, 5]))




```python

```