semvec/pypi-packages huggingface dataset:
keras-adaptive-softmax 0.9.0
adaptive-softmax implemented in Keras
License: MIT

# Keras Adaptive Softmax

[![Version](https://img.shields.io/pypi/v/keras-adaptive-softmax.svg)](https://pypi.org/project/keras-adaptive-softmax/)
![License](https://img.shields.io/pypi/l/keras-adaptive-softmax.svg)

\[[中文](https://github.com/CyberZHG/keras-adaptive-softmax/blob/master/README.zh-CN.md)|[English](https://github.com/CyberZHG/keras-adaptive-softmax/blob/master/README.md)\]

## Install

```bash
pip install keras-adaptive-softmax
```

## Usage

Generally, `AdaptiveEmbedding` and `AdaptiveSoftmax` should be used together. `AdaptiveEmbedding` provides variable length embeddings, while `AdaptiveSoftmax` calculates the similarities between the outputs and the generated embeddings.

```python
import keras
from keras_adaptive_softmax import AdaptiveEmbedding, AdaptiveSoftmax

input_layer = keras.layers.Input(shape=(None,))
embed_layer = AdaptiveEmbedding(
    input_dim=30,
    output_dim=32,
    cutoffs=[5, 15, 25],
    div_val=2,
    return_embeddings=True,
    return_projections=True,
    mask_zero=True,
)(input_layer)
dense_layer = keras.layers.Dense(
    units=32,
    activation='tanh',
)(embed_layer[0])
softmax_layer = AdaptiveSoftmax(
    input_dim=32,
    output_dim=30,
    cutoffs=[5, 15, 25],
    div_val=2,
    bind_embeddings=True,
    bind_projections=True,
)([dense_layer] + embed_layer[1:])
model = keras.models.Model(inputs=input_layer, outputs=softmax_layer)
model.compile('adam', 'sparse_categorical_crossentropy')
model.summary()
```

`cutoffs` and `div_val` controls the length of embeddings for each token. Suppose we have 30 distinct tokens, in the above example:

* The lengths of the embeddings of the first 5 tokens are 32
* The lengths of the embeddings of the next 10 tokens are 16
* The lengths of the embeddings of the next 10 tokens are 8
* The lengths of the embeddings of the last 5 tokens are 4
keras-rectified-adam 0.20.0
RAdam implemented in Keras & TensorFlow
License: MIT

# Keras RAdam

[![Version](https://img.shields.io/pypi/v/keras-rectified-adam.svg)](https://pypi.org/project/keras-rectified-adam/)
![License](https://img.shields.io/pypi/l/keras-rectified-adam.svg)

\[[中文](https://github.com/CyberZHG/keras-radam/blob/master/README.zh-CN.md)|[English](https://github.com/CyberZHG/keras-radam/blob/master/README.md)\]

Unofficial implementation of [RAdam](https://arxiv.org/pdf/1908.03265v1.pdf) in Keras. 

## Install

```bash
pip install keras-rectified-adam
```

## External Link

- [tensorflow/addons:RectifiedAdam](https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/rectified_adam.py)

## Usage

```python
from tensorflow import keras
import numpy as np
from keras_radam import RAdam

# Build toy model with RAdam optimizer
model = keras.models.Sequential()
model.add(keras.layers.Dense(input_shape=(17,), units=3))
model.compile(RAdam(), loss='mse')

# Generate toy data
x = np.random.standard_normal((4096 * 30, 17))
w = np.random.standard_normal((17, 3))
y = np.dot(x, w)

# Fit
model.fit(x, y, epochs=5)
```

### Use Warmup

```python
from keras_radam import RAdam

RAdam(total_steps=10000, warmup_proportion=0.1, min_lr=1e-5)
```