torch-cluster-based-norm 1.0.2
Cluster-Based Normalization provides versatile normalization layers for deep neural networks, including Supervised, Tiny Supervised, and Unsupervised versions. Enhance model generalization and robustness by efficiently integrating prior knowledge.
Requires Python: >=3.9

# Cluster-Based Normalization with PyTorch

Cluster-Based Normalization (CB-Norm) is a set of normalization layers designed for neural networks, accommodating both supervised and unsupervised learning scenarios. These layers aim to adaptively normalize data based on prior information, which can be beneficial in various machine learning tasks.

The Supervised versions (SCB-Norm and SCB-Norm-Base) are designed for scenarios where prior knowledge (clusters) is available and needs to be considered during normalization. These clusters, defined by experts or derived from clustering algorithms, can include classes, superclasses, or domains in domain adaptation scenarios.


## References


- **All versions:** *Cluster-Based Normalization Layer for Neural Networks*, FAYE et al., [ArXiv Link](https://arxiv.org/abs/2403.16798)


## Installation

To install the Cluster-Based Normalization package with **Pytorch** via pip, use the following command::

```bash
pip install torch-cluster-based-norm
```

## Usage

### Dataset

```python
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

# Create data
data = np.array([[1, 2, 3, 4, 5],
                 [6, 7, 8, 9, 10],
                 [11, 12, 13, 14, 15],
                 [16, 17, 18, 19, 20],
                 [21, 22, 23, 24, 25],
                 [26, 27, 28, 29, 30],
                 [31, 32, 33, 34, 35],
                 [36, 37, 38, 39, 40],
                 [41, 42, 43, 44, 45],
                 [46, 47, 48, 49, 50]])

X = torch.tensor(data, dtype=torch.float32)

# Create target (5 classes)
labels = [0, 1, 2, 3, 4, 0, 1, 2, 3, 4]  
Y = torch.tensor(labels, dtype=torch.long)


# Establishing clusters (3 clusters): SCBNorm employs indices as input for normalizing, while SCBNormBase utilizes a one-hot representation of indices as input.
cluster_indices = [0, 1, 2, 0, 1, 2, 0, 1, 2, 0] 
cluster_scb_norm = torch.tensor(cluster_indices)
cluster_scb_norm_base = torch.tensor(np.eye(3)[cluster_indices], dtype=torch.float32)

```


### Tiny Supervised Cluster-Based Normalization (SCBNormBase)

SCBNormBase isa soft version of Supervised Cluster-Based Normalization. It also requires data and cluster as input, but it provides a lighter approach compared to Supervised Cluster-Based Normalization.

```python
from torch_cluster_based_norm import SCBNormBase

# Apply normalization layer
scb_layer = SCBNormBase(num_clusters=3, input_dim=5)

# Define the rest of your model architecture
# For example:
hidden_layer = nn.Linear(5, 10)
output_layer = nn.Linear(10, 10)

# Define the model
model = nn.Sequential(
    scb_layer,
    nn.ReLU(),
    hidden_layer,
    nn.ReLU(),
    output_layer
)

# Define loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train the model
epochs = 10
for epoch in range(epochs):
    optimizer.zero_grad()
    output = model([X, cluster_scb_norm_base])
    loss = criterion(output, Y)
    loss.backward()
    optimizer.step()
    print(f'Epoch [{epoch+1}/{epochs}], Loss: {loss.item()}')

```

### Supervised Cluster-Based Normalization (SCBNorm)

SCBNorm adapts the normalization process based on prior information (cluster) provided alongside the input data. By incorporating such cluster, the layer enhances the normalization process, leading to accelerate convergence and improved performance.

```python
from torch_cluster_based_norm import SCBNorm

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.hidden_layer = nn.Linear(5, 10)
        self.scb_norm = SCBNorm(num_clusters=3, input_dim=10)  
        self.output_layer = nn.Linear(10, 5)

    def forward(self, x, cluster_id):
        hidden_output = self.hidden_layer(x)
        normalized_activation = self.scb_norm((hidden_output, cluster_id))
        output = self.output_layer(normalized_activation)
        return output

# Instantiate the model
model = MyModel()

# Define optimizer and loss function
optimizer = optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss()


# Training loop
num_epochs = 10
for epoch in range(num_epochs):
    # Forward pass
    outputs = model(X, cluster_scb_norm)
    loss = criterion(outputs, Y)

    # Backward pass and optimization
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Print loss for monitoring training progress
    print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}")

```

### Unsupervised Cluster-Based Normalization (UCB-Norm)

This version doesn't require explicit prior information and adapts based on the input data distribution.

```python
from torch_cluster_based_norm import UCBNorm

# Define the model
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.ucb_norm = UCBNorm(num_clusters=3, input_dim=5)
        self.hidden_layer = nn.Linear(5, 10)
        self.output_layer = nn.Linear(10, 5)

    def forward(self, x):
        x = self.ucb_norm(x)
        x = F.relu(self.hidden_layer(x))
        x = F.softmax(self.output_layer(x), dim=1)
        return x

# Instantiate the model
model = MyModel()

# Define optimizer and loss function
optimizer = optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss()


# Training loop
num_epochs = 10
for epoch in range(num_epochs):
    # Forward pass
    outputs = model(X)
    loss = criterion(outputs, Y)

    # Backward pass and optimization
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Print loss for monitoring training progress
    print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}")


```


This README provides an overview of the Cluster-Based Normalization package along with examples demonstrating the usage of different normalization layers. You can modify and extend these examples according to your specific requirements.