semvec/pypi-packages huggingface dataset:
rezero 0.1.0
ReZero networks
Requires Python: >=3.6

# ReZero for Deep Neural Networks

[**ReZero is All You Need: Fast Convergence at Large Depth**](https://arxiv.org/abs/2003.04887); *ArXiv, March 2020*.

Thomas Bachlechner*, Bodhisattwa Prasad Majumder*, Huanru Henry Mao*, Garrison W. Cottrell, Julian McAuley (* denotes equal contributions)

This repository contains the ReZero-Transformer implementation from the paper. It matches Pytorch's Transformer and can be easily used as a drop-in replacement. See sections below for installation and usage.

# Abstract

Deep networks have enabled significant performance gains across domains, but they often suffer from vanishing/exploding gradients. This is especially true for Transformer architectures where depth beyond 12 layers is difficult to train without large datasets and computational budgets. In general, we find that inefficient signal propagation impedes learning in deep networks. In Transformers, multi-head self-attention is the main cause of this poor signal propagation. To facilitate deep signal propagation, we propose **ReZero**, a simple change to the architecture that initializes an arbitrary layer as the identity map, using a single additional learned parameter per layer. We apply this technique to language modeling and find that we can easily train ReZero-Transformer networks over a hundred layers. When applied to 12 layer Transformers, ReZero converges 56% faster on enwiki8. ReZero applies beyond Transformers to other residual networks, enabling 1,500% faster convergence for deep fully connected networks and 32% faster convergence for a ResNet-56 trained on CIFAR 10.

# Installation
Simply install from pip:

```
pip install rezero
```

Pytorch 1.4 or greater is required.

# Usage
We provide custom ReZero Transformer layers (RZTX).

For example, this will create a Transformer encoder:
```py
import torch
import torch.nn as nn
from rezero.transformer import RZTXEncoderLayer

encoder_layer = RZTXEncoderLayer(d_model=512, nhead=8)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
src = torch.rand(10, 32, 512)
out = transformer_encoder(src)
```

This will create a Transformer decoder:
```py
import torch
import torch.nn as nn
from rezero.transformer import RZTXDecoderLayer

decoder_layer = RZTXDecoderLayer(d_model=512, nhead=8)
transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6)
memory = torch.rand(10, 32, 512)
tgt = torch.rand(20, 32, 512)
out = transformer_decoder(tgt, memory)
```

Make sure `norm` argument is left as `None` as to not use `LayerNorm` in the Transformer.

See https://pytorch.org/docs/master/nn.html#torch.nn.Transformer for details on how to integrate customer Transformer layers to Pytorch.
YALogger 1.3
Yet Another custom logger
Requires Python: >=3.7

# Yet Another Logger    [![Documentation Status](https://readthedocs.org/projects/yalogger/badge/?version=latest)](https://yalogger.readthedocs.io/en/latest/?badge=latest) ![PyPI](https://img.shields.io/pypi/v/YALogger) [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://github.com/DivyenduDutta/YALogger/blob/master/LICENSE.md) [![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)

Minimalist logger written in python.

Currently only provides three functions for logging. Also currently log() only supports logging text and dict, whereas
for other python objects it logs the type only. Enhancement open in issues.

1. **Method entry logging**

2. **Method exit logging**

3. **Normal logging**

Both Method entry logging and Method exit logging are only *INFO* level logging by default

Supports *3 levels of logging* - *INFO, ERROR, DEBUG*. These are mutually exclusive (ie not hierarchial)

Supports *2 modes of logging*:

1. **FILE** - Writes logs to a file in the logs folder 

2. **CONSOLE** - Logs to the standard output console

Uploaded to PIP - [PIP repo](https://pypi.org/project/YALogger).


**Log format - [log_level timestamp] [Module name]-[Method name] <log text>**

## Documentation

Hosted on [Read The Docs](https://yalogger.readthedocs.io/en/latest/).

Achieved via Sphinx (as the doc build tool), reStructuredText as the markup and hosted on Read The Docs website.

## INSTALLATION

run 
```
pip install YALogger
```

## IMAGE
Log file looks like source

![alt text](https://github.com/DivyenduDutta/YALogger/blob/master/images/log%20output%20file.PNG)

logger.properties looks like so

![alt text](https://github.com/DivyenduDutta/YALogger/blob/master/images/logger%20properties.PNG)


## REQUIREMENTS

1. logger.properties file

2. logs folder


logger.properties file needs to have **[logger properties]** at the root

## SAMPLE USAGE

```from YALogger.custom_logger import Logger 
Logger.initialize_logger(logger_prop_file_path = '.\logger.properties',log_file_path = './logs') 
Logger.perform_method_entry_logging('foo','bar')
Logger.perform_method_exit_logging('foo','bar') 
Logger.log('info', 'foo','bar','this is the log text')
```

Look for sample usage code in sample folder

Refer to [sample logger.properties](https://github.com/DivyenduDutta/YALogger/blob/master/Yet%20Another%20Logger/sample/logger.properties).

Ensure logs folder is present 


## License

This is an open source tool licensed under GPL v3.0. Copy of the license can be found
[here](https://github.com/DivyenduDutta/goodreads-scraper-visualizer/blob/master/LICENSE.md).