semvec/pypi-packages huggingface dataset:
naren-api 0.0.5
A small python package
License: MIT
Requires Python: >=3.9

## ::: Base Skeleton for Data-Science Projects :::

### Steps to run this Bare-Bones Template: 
**{Note}**: Operates exclusively within the 'Git Bash' environment.

1. Run below command 
```bash
python template.py
```
-|- Upon initiating the project, remove the 'temp.log' temporary log file.

-|- Include modules or packages in the requirements.txt file according to the specific requirements of your project.

-|- Modify the 'setup.cfg' and 'setup.py' files according to the specific requirements(project_name_description) of the project.

2. Run below command
```bash
bash init_setup.sh
```

**To activate environment in bash:**
```bash
source ~/Anaconda3/etc/profile.d/conda.sh
conda activate venv/
```

3. Run below command (optional)
```bash
pytest -v
tox
```
<br>

---

The base skeleton for a data science project typically includes the following components:

1. **Project Structure:**
   - Create a well-organized directory structure for your project.
   - Divide folders for data, code, documentation, and models.

2. **Data Collection and Exploration:**
   - Gather relevant data from various sources.
   - Explore and understand the data through descriptive statistics and visualizations.

3. **Data Cleaning and Preprocessing:**
   - Handle missing values, outliers, and inconsistencies in the data.
   - Transform and preprocess the data for analysis.

4. **Feature Engineering:**
   - Create new features or transform existing ones to improve model performance.

5. **Model Building:**
   - Select appropriate algorithms based on the problem.
   - Split the data into training and testing sets.
   - Train and evaluate models using appropriate metrics.

6. **Model Deployment:**
   - Deploy the chosen model for real-world use if applicable.

7. **Documentation:**
   - Provide clear documentation for your code, including comments and a README file.

8. **Version Control:**
   - Use version control (e.g., Git) to track changes and collaborate with others.

9. **Testing and Validation:**
   - Implement testing procedures to validate the correctness of your code.

10. **Visualization and Reporting:**
    - Create visualizations to communicate insights.
    - Prepare a report summarizing the findings and methodology.

This basic structure ensures a systematic and reproducible approach to data science projects.
givt-pytorch 0.0.3
A partial implementation of Generative Infinite Vocabulary Transformer (GIVT) from Google Deepmind, in PyTorch.
License: MIT
Requires Python: >=3.9

## GIVT-PyTorch
A partial implementation of [Generative Infinite Vocabulary Transformer (GIVT)](https://arxiv.org/abs/2312.02116) from Google Deepmind, in PyTorch.

This repo only implements the causal version of GIVT, and does away with the k mixtures predictions or the use of the full covariance matrix, as for most purposes they did not yield significantly better results.

The decoder transformer implementation is also modernized, adopting a Llama style architecture with gated MLPs, SiLU, RMSNorm, and RoPE.

## Install
```sh
# for inference
pip install .

# for training/development
pip install -e '.[train]'
```


## Usage
```py
from givt_pytorch import GIVT

# load pretrained checkpoint
model = GIVT.from_pretrained('elyxlz/givt-test')

latents = torch.randn((4, 500, 32)) # vae latents (bs, seq_len, size)
loss = model.forward(latents).loss # NLL Loss

prompt = torch.randn((50, 32)) # no batched inference implemented
generated = model.generate(
    prompt=prompt, 
    max_len=500,
    cfg_scale=0.5,
    temperature=0.95,
) # (500, 32)
```

## Training

Define a config file in `configs/`, such as this one:
```py
from givt_pytorch import (
    GIVT,
    GIVTConfig,
    DummyDataset,
    Trainer,
    TrainConfig
)

model = GIVT(GIVTConfig())
dataset = DummyDataset()

trainer = Trainer(
    model=model,
    dataset=dataset,    
    train_config=TrainConfig()
)
```
Create an accelerate config.
```sh
accelerate config
```

And then run the training.
```sh
accelerate launch train.py {config_name}
```

## TODO
- [ ] Test out with latents from an audio vae
- [ ] Add CFG with rejection sampling

## References
```bibtex
@misc{litgpt2024,
  title={lit-gpt on GitHub},
  url={https://github.com/Lightning-AI/lit-gpt},
  year={2024}

@misc{tschannen2023givt,
    title   = {GIVT: Generative Infinite-Vocabulary Transformers}, 
    author  = {Michael Tschannen, Cian Eastwood, Fabian Mentzer},
    year    = {2023},
    eprint  = {2312.02116},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}
```