pytortto 0.0.2
A PyTorch style machine learning framework written in NumPy/CuPy with GPU acceleration.
Requires Python: >=3.6

## PyTortto
<p align="center">
  <img src="https://github.com/samrere/Pytorch-from-scratch/blob/main/pics/tortto.png" height="300">
</p> 

## Intro
This is a toy machine learning framework implemented in numpy, using pytorch syntax.

Similar to the pokemon "ditto", pytortto (pronounced as "py-turtle") works exactly like pytorch, although inferior in terms of speed, because its core is written entirely in numpy. The purpose of this project is to understand how pytorch, or how general machine learning algorithms work under the hood. Max effort was given to correctness, calculation efficiency (like simpler Jacobian in logsoftmax, efficient implementation of convolution etc.), numerical stability (log-sum-exp used in logsigmoid, BCEWithLogitsLoss etc.), and memory efficiency (implementation of caching, view etc.).  

Tortto implements reverse mode automatic differentiation and supports dynamic computation graph like pytorch.

Tortto is strict with tensor dimensionality: every tensor should have dim>=2 except views. Numbers are treated as 1x1 matrix `[[1]]`, 1D array `[1,2,3]` should be converted to a 2d row vector `[[1,2,3]]`, column vectors should be written as `[[1],[2],[3]]`. Parameters that are 1d arrays in pytorch, like bias in Linear and in Conv2d, are also reshaped to 1xN row vector and Nx1x1 tensor respectively.

## Prerequisites
**Numpy** (Required): Only use its basic functions. The "highest-level" function used are FFT, but I also implemented those in another repo: [FFT from scratch](https://github.com/samrere/fft-from-scratch). To use FFT in `Conv2d` and `ConvTranspose2d`, set `os.environ['fft'] = 'True'` before `import tortto`  

**Scipy** (Optional): Only use 2 functions: `scipy.fft.rfft2` and `scipy.fft.irfft2`, their Numpy equivalence will be used if Scipy is not installed, but Numpy FFT upcast everyting to Complex128 which is undesirable for float32 inputs.  

**Cupy** (Optional): compute Numpy functions in GPU. set `os.environ['cuda'] = 'True'` before `import tortto`  

## Examples
**pytortto uses the same syntax as pytorch and produces the same result. Here shows the forward and backward propagation of some functions:**

**pytortto**
```python
import tortto as tt

x = tt.tensor([[1.1, 2.0, 3.6], [4.7, 3.14, 2.718]], requires_grad=True)
y = tt.tensor([[2., 3.99], [8.4, 1.5], [2.5, 7.8]], requires_grad=True)
output = tt.tanh(1/(tt.cat([tt.sin((tt.exp(x ** 1.4) / 3.1 ** tt.log(x)) @ y), y]).var(axis=0, unbiased=False).mean()))
output.backward()

print(output) # every number in tortto is a 1x1 matrix
# tensor([[0.1168]], grad_fn=<tanhBackward>)
print(x.grad)
# [[  0.02126594  -0.37090623   8.55691   ]
#  [-10.720857     2.7745626   -0.4861354 ]]
print(y.grad)
# [[ 5.5675573  -3.9194558 ]
#  [ 0.14517403 -0.09022781]
#  [-0.55190086  0.6685495 ]]
```
**compare with pytorch**
```python
import torch

x = torch.tensor([[1.1, 2.0, 3.6], [4.7, 3.14, 2.718]], requires_grad=True)
y = torch.tensor([[2., 3.99], [8.4, 1.5], [2.5, 7.8]], requires_grad=True)
output = torch.tanh(1/(torch.cat([torch.sin((torch.exp(x ** 1.4) / 3.1 ** torch.log(x)) @ y), y]).var(axis=0, unbiased=False).mean()))
output.backward()

print(output)
# tensor(0.1168, grad_fn=<TanhBackward0>)
print(x.grad)
# tensor([[  0.0213,  -0.3709,   8.5569],
#         [-10.7209,   2.7746,  -0.4861]])
print(y.grad)
# tensor([[ 5.5676, -3.9195],
#         [ 0.1452, -0.0902],
#         [-0.5519,  0.6685]])
```
<br /><br />
**Tortto also support forward and backward propagation of common modules like Conv2d, ConvTranspose2d, BatchnormNd, ReLU, Softmax, NLLLoss etc., optimizers including SGD and Adam, and lr_schedulers, which can then be combined to train some small models:**

### **[preact resnet56 on CIFAR10](https://github.com/samrere/Pytorch-from-scratch/blob/main/examples/resnet/resnet_CIFAR10.ipynb)**  
#.filters | n | batchsize | train/val/test | epochs | test error
--- | --- | --- | --- | --- | --- |
(16,32,64) | (9,9,9) | 128 | 45k/5k/10k | 180 | 6.53%

* memory: ~2.3GB
* training time: ~6 hours on google cloud with a Tesla T4, ~18 hours on my laptop with a GTX965M
* compared with pytorch: ~1.5 hours on a Tesla T4, with cuda and cudnn.benchmark on  

loss             |  error
:-------------------------:|:-------------------------:
![](https://github.com/samrere/Pytorch-from-scratch/blob/main/pics/loss.png)  |  ![](https://github.com/samrere/Pytorch-from-scratch/blob/main/pics/error.png)  
<p align="center">
  <img src="https://github.com/samrere/Pytorch-from-scratch/blob/main/pics/cifar10.png" width="800">
</p>  

### **[UNet segmentation on carvana dataset](https://github.com/samrere/Pytorch-from-scratch/blob/main/examples/unet/UNet.ipynb)**  
image size | #.filters | batchsize | train/val/test | epochs
--- | --- | --- | --- | --- |
resized to 3x64x64 | 32,64,128,256,512,256,128,64,32 | 32 | 3658/646/784 | 20
* memory: ~1.9GB
* training time: ~7.5 mins on a Tesla T4, ~23 mins on a GTX965M  

<p align="center">
  <img src="https://github.com/samrere/Pytorch-from-scratch/blob/main/pics/carvana.png" width="800">
</p>  

## Supported functionalities
### Misc. 
#### variable functions
`manual_seed`, `clear`, `from_numpy`, `ones`, `ones_like`, `zeros`, `zeros_like`, `randn`, `eye`, `arange`, `linspace`, `tensor`   
#### transforms
`gaussian_kernel`, `Compose`, `ToTensor`, `Normalize`   

### Autograd, forward and backward propagations of:
#### basic operations [tortto/autograd/grad_fcn.py](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/autograd/grad_fcn.py)   
add: `+`, sub: `-`, mul: `*`, div: `/`, `sum`, `mean`, `var`      

#### ufunc [tortto/autograd/grad_ufunc.py](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/autograd/grad_ufunc.py)   
`exp`, `log`, `sin`, `cos`, `tanh`, `sigmoid`, power: `x**2`, exponential: `2**x`,

#### tensor [tortto/autograd/grad_fcn.py](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/autograd/grad_fcn.py)   
matmul: `@`, elementwise (Hadamard) mul: `*`, transpose: `.T` or `transpose(x, axes)`, concatenate: `cat`, view: `view`, slicing: `x[...,1:10:2,:]`, `flatten`      

#### nn.modules [tortto/nn/functional.py](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/nn/functional.py)   
* activation: `Tanh`, `Sigmoid`, `LogSigmoid`, `ReLU`, `Softmax`, `LogSoftmax`  
* batchnorm: `BatchNorm1d`, `BatchNorm2d`, `BatchNorm3d`   
* container: `Sequential`, `ModuleList`  
* conv: `Conv2d`  
* transposed conv: `ConvTranspose2d`
* dropout: `Dropout`    
* linear: `Linear`    
* loss: `MSELoss`, `BCELoss`, `BCEWithLogitsLoss`, `NLLLoss`  
* module: `Module`  
* pooling: `MaxPool2d`    

#### optim [tortto/optim/_functional.py](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/optim/_functional.py)     
adapted from pytorch code, since it's already written in python
* optimizer: `Optimizer`   
* sgd: `SGD`
* Adam: `Adam`
* [lr_scheduler](https://github.com/samrere/Pytorch-from-scratch/blob/main/tortto/optim/lr_scheduler.py): `LambdaLR`, `StepLR`, `MultiStepLR`, `CosineAnnealingLR`