File size: 754 Bytes
35d7048 30fac58 35d7048 dfc9313 35d7048 5a008cd 974f8dd fff1ccb 974f8dd fff1ccb 974f8dd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
---
language:
- en
license: mit
tags:
- Kolmogorov-Arnold Network
- BerKANT
- KAN
---
# BerKANT (training)
A Bert implementation where most of the `torch.nn.linear` have been replaced with `KANLinear`.
Currently pretraining on [JackBAI/bert_pretrain_datasets](https://huggingface.co/datasets/JackBAI/bert_pretrain_datasets) on a RTX 4090. Will be do in 5 days from 13/05/2024. Until then :)
```python
from transformers import AutoModelForMaskedLM, AutoConfig
# Define the model path
model_path = "isemmanuelolowe/BerKANT_171M"
# Load the configuration
config = AutoConfig.from_pretrained(model_path)
# Load the model with the correct configuration
model = AutoModelForMaskedLM.from_pretrained(model_path, config=config, trust_remote_code=True)
``` |