semvec/pypi-packages huggingface dataset:
keras-pos-embd 0.13.0
Position embedding layers in Keras
License: MIT

# Keras Position Embedding

[![Version](https://img.shields.io/pypi/v/keras-pos-embd.svg)](https://pypi.org/project/keras-pos-embd/)

\[[中文](https://github.com/CyberZHG/keras-pos-embd/blob/master/README.zh-CN.md)|[English](https://github.com/CyberZHG/keras-pos-embd/blob/master/README.md)\]

Position embedding layers in Keras.

## Install

```bash
pip install keras-pos-embd
```

## Usage

### Trainable Embedding

```python
from tensorflow import keras
from keras_pos_embd import PositionEmbedding

model = keras.models.Sequential()
model.add(PositionEmbedding(
    input_shape=(None,),
    input_dim=10,     # The maximum absolute value of positions.
    output_dim=2,     # The dimension of embeddings.
    mask_zero=10000,  # The index that presents padding (because `0` will be used in relative positioning).
    mode=PositionEmbedding.MODE_EXPAND,
))
model.compile('adam', 'mse')
model.summary()
```

Note that you don't need to enable `mask_zero` if you want to add/concatenate other layers like word embeddings with masks:

```python
from tensorflow import keras
from keras_pos_embd import PositionEmbedding

model = keras.models.Sequential()
model.add(keras.layers.Embedding(
    input_shape=(None,),
    input_dim=10,
    output_dim=5,
    mask_zero=True,
))
model.add(PositionEmbedding(
    input_dim=100,
    output_dim=5,
    mode=PositionEmbedding.MODE_ADD,
))
model.compile('adam', 'mse')
model.summary()
```

### Sin & Cos Embedding

The [sine and cosine embedding](https://arxiv.org/pdf/1706.03762) has no trainable weights. The layer has three modes, it works just like `PositionEmbedding` in `expand` mode:

```python
from tensorflow import keras
from keras_pos_embd import TrigPosEmbedding

model = keras.models.Sequential()
model.add(TrigPosEmbedding(
    input_shape=(None,),
    output_dim=30,                      # The dimension of embeddings.
    mode=TrigPosEmbedding.MODE_EXPAND,  # Use `expand` mode
))
model.compile('adam', 'mse')
model.summary()
```

If you want to add this embedding to existed embedding, then there is no need to add a position input in `add` mode:

```python
from tensorflow import keras
from keras_pos_embd import TrigPosEmbedding

model = keras.models.Sequential()
model.add(keras.layers.Embedding(
    input_shape=(None,),
    input_dim=10,
    output_dim=5,
    mask_zero=True,
))
model.add(TrigPosEmbedding(
    output_dim=5,
    mode=TrigPosEmbedding.MODE_ADD,
))
model.compile('adam', 'mse')
model.summary()
```
keras-layer-normalization 0.16.0
Layer normalization implemented in Keras
License: MIT

# Keras Layer Normalization

[![Version](https://img.shields.io/pypi/v/keras-layer-normalization.svg)](https://pypi.org/project/keras-layer-normalization/)
![License](https://img.shields.io/pypi/l/keras-layer-normalization.svg)

Implementation of the paper: [Layer Normalization](https://arxiv.org/pdf/1607.06450.pdf)

## Install

```bash
pip install keras-layer-normalization
```

## Usage

```python
from tensorflow import keras
from keras_layer_normalization import LayerNormalization


input_layer = keras.layers.Input(shape=(2, 3))
norm_layer = LayerNormalization()(input_layer)
model = keras.models.Model(inputs=input_layer, outputs=norm_layer)
model.compile(optimizer='adam', loss='mse', metrics={},)
model.summary()
```