feat: add application file

.gitignore

@@ -0,0 +1,162 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+weights/
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

README.md

@@ -1,14 +1,191 @@
----
-title: QuillGPT
-emoji: 📉
-colorFrom: yellow
-colorTo: yellow
-sdk: streamlit
-sdk_version: 1.40.2
-app_file: app.py
-pinned: false
-license: mit
-short_description: Implementation of the GPT decoder block in PyTorch
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+![QuillGPT-cropped-removebg-preview](https://github.com/NotShrirang/QuillGPT/assets/85283622/2e63d8ce-24f8-4bf0-835a-0c621f1d7400)
+
+# QuillGPT
+
+![GitHub stars](https://img.shields.io/github/stars/NotShrirang/GPT-From-Scratch?style=social)
+![GitHub forks](https://img.shields.io/github/forks/NotShrirang/GPT-From-Scratch?style=social)
+![GitHub commits](https://img.shields.io/github/commit-activity/t/NotShrirang/QuillGPT)
+![GitHub issues](https://img.shields.io/github/issues/NotShrirang/GPT-From-Scratch)
+![GitHub pull requests](https://img.shields.io/github/issues-pr/NotShrirang/GPT-From-Scratch)
+![GitHub](https://img.shields.io/github/license/NotShrirang/GPT-From-Scratch)
+![GitHub last commit](https://img.shields.io/github/last-commit/NotShrirang/GPT-From-Scratch)
+![GitHub repo size](https://img.shields.io/github/repo-size/NotShrirang/GPT-From-Scratch)
+![Streamlit Playground](https://img.shields.io/badge/Streamlit%20App-red?style=flat-rounded-square&logo=streamlit&labelColor=white)
+![Docker Container](https://img.shields.io/badge/docker-blue?style=flat-rounded-square&logo=docker&labelColor=white)
+
+QuillGPT is an implementation of the GPT decoder block based on the architecture from [Attention is All You Need](https://arxiv.org/abs/1706.03762) paper by Vaswani et. al. implemented in PyTorch. Additionally, this repository contains two pre-trained models—Shakespearean GPT and Harpoon GPT—along with their trained weights. For ease of experimentation and deployment, a Streamlit Playground is provided for interactive exploration of these models and FastAPI microservice implemented with Docker containerization for scalable deployment. You'll also find Python scripts for training new GPT models and performing inference on them, along with notebooks showcasing trained models. To facilitate text encoding and decoding, a simple tokenizer is implemented. Explore QuillGPT to utilize these tools and enhance your natural language processing projects!
+
+## Table of Contents
+
+- [Models](#models)
+- [Getting Started](#getting-started)
+  - [Installation](#installation)
+  - [Streamlit Playground](#streamlit-playground)
+  - [FastAPI Microservice](#for-running-fastapi-microservice)
+  - [Running Docker Container](#for-using-containerized-version)
+- [Usage](#usage)
+  - [Training the GPT Model](#training-the-gpt-model)
+  - [Using the Trained Model for Inference](#for-inference)
+- [Explanation](#explanation)
+  - [Decoder Block](#the-decoder-block)
+  - [Input Embeddings](#input-embeddings)
+  - [Positional Embeddings](#positional-embeddings)
+  - [Self-Attention](#self-attention)
+- [License](#license)
+- [Contributing](#contributing)
+- [Support](#support)
+
+## <div align="center">Models</div>
+
+There are two pre-trained models and weights included in this repository.
+
+| Feature                        | Shakespearean GPT                                                                                                     | Harpoon GPT                                                                                                                                  |
+| ------------------------------ | --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Parameters**                 | 10.7 M                                                                                                                | 226 M                                                                                                                                        |
+| **Weights**                    | [Weights](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/weights/GPT_model_char.pt)                        | [Weights](https://www.dropbox.com/scl/fi/vi5z3s17otn0jf7sr40po/Harpoon_Corpus_GPT_model.pt?rlkey=r7oppeslusv736fzmi908le95&st=wak0uf2t&dl=0) |
+| **Model Config**               | [Config](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/config/shakespearean_config.json)                  | [Config](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/config/harpoon_config.json)                                               |
+| **Training Data**              | Text from Shakespearean plays ([input.txt](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/data/input.txt)) | Random text from books ([corpus.txt](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/data/corpus.txt))                             |
+| **Embedding Type**             | Character embeddings                                                                                                  | Character embeddings                                                                                                                         |
+| **Training Notebook**          | [Notebook](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/notebooks/GPT_From_Scratch_CharEmbeddings.ipynb) | [Notebook](https://github.com/NotShrirang/GPT-From-Scratch/blob/main/notebooks/GPT_From_Scratch_with_1024_char_embd.ipynb)                   |
+| **Hardware**                   | NVIDIA T4                                                                                                             | NVIDIA A100                                                                                                                                  |
+| **Training & Validation Loss** | ![loss](https://github.com/user-attachments/assets/df89c1f6-d89a-4a3a-8340-edcf7416878c)                              | ![loss](https://github.com/user-attachments/assets/76c5e0d1-a53c-4d0d-ac8f-5529ec3a5008)                                                     |
+
+## Getting Started:
+
+### Installation:
+
+To run the training and inference scripts, follow these steps:
+
+1. Clone the repository:
+
+```sh
+git clone https://github.com/NotShrirang/GPT-From-Scratch.git
+cd GPT-From-Scratch
+```
+
+2. Install the required packages:
+
+```sh
+pip install -r requirements.txt
+```
+
+Make sure you download the weights for Harpoon GPT from [here](https://www.dropbox.com/scl/fi/vi5z3s17otn0jf7sr40po/Harpoon_Corpus_GPT_model.pt?rlkey=r7oppeslusv736fzmi908le95&st=wak0uf2t&dl=0) before proceeding!
+
+### Streamlit Playground:
+
+It is hosted on Streamlit Cloud Service. You can visit it through the link [here](https://quillgpt.streamlit.app/).
+
+[![Streamlit Demo](https://github.com/NotShrirang/GPT-From-Scratch/assets/85283622/fa888670-2c44-4f97-a07d-c58473d847d0)](https://quillgpt.streamlit.app/)
+
+```sh
+streamlit run app.py
+```
+
+### For running FastAPI Microservice:
+
+```sh
+python main.py
+```
+
+### For using Containerized Version:
+
+#### Build and Run the Docker Container with bash:
+
+```sh
+./run.sh start-dev
+```
+
+#### To stop the Docker Container, run the following command:
+
+```sh
+./run.sh stop-dev
+```
+
+## Usage
+
+### Training the GPT Model:
+
+To train the GPT model, follow these steps:
+
+1. Prepare data. Put the whole text data into single .txt file and save it.
+2. Write the configurations for transformer and save the file.
+   <br>For example:
+   `json
+ {
+   "data_path": "data/corpus.txt",
+   "vocab_size": 135,
+   "batch_size": 32,
+   "block_size": 256,
+   "max_iters": 3000,
+   "eval_interval": 300,
+   "learning_rate": 3e-5,
+   "eval_iters": 50,
+   "n_embd": 1024,
+   "n_head": 12,
+   "n_layer": 18,
+   "dropout": 0.3,
+ }
+ `
+
+3. Train model using script `scripts/train_gpt.py`
+
+```bash
+python scripts/train_gpt.py \
+        --config_path config/config.json \
+        --data_path data/corpus.txt \
+        --output_dir trained_models
+```
+
+(You can change the `config_path`, `data_path` and `output_dir` as per your requirements.)
+
+4. The trained model will be saved in the `output_dir` specified in the command.
+
+### For Inference:
+
+After training, you can use the trained GPT model for text generation. Here's an example of using the trained model for inference:
+
+```bash
+python scripts/inference_gpt.py \
+        --config_path config/shakespearean_config.json \
+        --weights_path weights/GPT_model_char.pt \
+        --max_length 500 \
+        --prompt "Once upon a time"
+```
+
+## <div align="center">Explanation</div>
+
+### The Decoder Block:
+
+<div align="center"><img src="https://github.com/NotShrirang/GPT-From-Scratch/assets/85283622/397049a3-10cc-49b5-8696-f19806b2668e" width=350 alt="Decoder Architecture"/></div>
+
+The decoder block is a crucial component of the GPT (Generative Pre-trained Transformer) model, it is where GPT actually generates the text. It leverages the self-attention mechanism to process input sequences and generate coherent outputs. Each decoder block consists of multiple layers, including self-attention layers, feed-forward neural networks, and layer normalization. The self-attention layers allow the model to weigh the importance of different words in a sequence, capturing context and dependencies regardless of their positions. This enables the GPT model to generate contextually relevant text.
+
+### Input Embeddings:
+
+<div align="center">![vector embeddings](https://github.com/NotShrirang/GPT-From-Scratch/assets/85283622/29b4c375-c9f0-47b9-9d34-2a21dfdf0be8)</div>
+
+Input embeddings play a crucial role in transformer-based models like GPT by transforming input tokens into meaningful numerical representations. These embeddings serve as the initial input for the model, capturing semantic information about the words in the sequence. The process involves mapping each token in the input sequence to a high-dimensional vector space, where similar tokens are positioned closer together. This enables the model to understand the relationships between different words and effectively learn from the input data. The input embeddings are then fed into the subsequent layers of the model for further processing.
+
+### Positional Embeddings:
+
+![positional_encoding](https://github.com/NotShrirang/GPT-From-Scratch/assets/85283622/90293fb0-8f20-4dc0-adba-8c31a54ef4f4)
+
+In addition to input embeddings, positional embeddings are another vital component of transformer architectures such as GPT. Since transformers lack inherent information about the order of tokens in a sequence, positional embeddings are introduced to provide the model with positional information. These embeddings encode the position of each token within the sequence, allowing the model to distinguish between tokens based on their positions. By incorporating positional embeddings, transformers like GPT can effectively capture the sequential nature of data and generate coherent outputs that maintain the correct order of words in the generated text.
+
+### Self-Attention:
+
+![self attention](https://github.com/NotShrirang/GPT-From-Scratch/assets/85283622/a6d785e4-ab00-4da0-a072-791f680d2bb8)
+
+Self-attention, a fundamental mechanism in transformer-based models like GPT, operates by assigning importance scores to different words in a sequence. This process involves three key steps: calculating attention scores, applying softmax to obtain attention weights, and finally combining these weights with the input embeddings to generate contextually informed representations. At its core, self-attention allows the model to focus more on relevant words while de-emphasizing less important ones, facilitating effective learning of contextual dependencies within the input data. This mechanism is pivotal in capturing long-range dependencies and contextual nuances, enabling transformer models to generate long sequences of text.
+
+## License
+
+MIT © [Shrirang Mahajan](https://github.com/NotShrirang)
+
+## Contributing
+
+Feel free to submit pull requests, create issues, or spread the word!
+
+## Support
+
+Support me by simply starring this repository! ⭐

app.py

@@ -0,0 +1,100 @@
+import torch
+import streamlit as st
+from colorama import Fore
+from core.models.gpt import GPTLanguageModel
+from core.tokenizers.tokenizer import Tokenizer
+from core.utils.gptutils import hyperparameters, load_data
+
+st.set_page_config(layout='wide',
+                   page_title='QuillGPT',
+                   page_icon='🪶',
+                   initial_sidebar_state='expanded'
+                   )
+
+def decode_text(input, model: GPTLanguageModel, max_tokens, temperature):
+    for idx in model.generate(idx=input, max_new_tokens=max_tokens, max_seq_length=50, temperature=temperature):
+        text = tokenizer.decode(idx[0].tolist())[-1]
+        yield text
+
+models = {
+    "Shakespearean GPT": './weights/GPT_model_char.pt',
+    "GPT": './weights/Harpoon_Corpus_GPT_model_word2.pt',
+}
+
+st.sidebar.header('QuillGPT')
+
+st.sidebar.write("This app generates text using a GPT model trained on either the Harpoon corpus or Shakespearean plays.")
+
+# Select one of the two model
+model_name = st.sidebar.selectbox('Select a model:', list(models.keys()))
+if model_name == "GPT":
+    st.title('GPT From Scratch')
+    st.write("This model was trained on the Harpoon corpus.")
+else:
+    st.title('Shakespearean GPT')
+    st.write("This model was trained on Shakespearean plays.")
+
+path = models[model_name]
+
+if model_name == "GPT":
+    config_path = './config/harpoon_config.json'
+    data_path = './data/corpus.txt'
+    name = "Harpoon GPT"
+    tokenizer: Tokenizer = Tokenizer()
+    tokenizer.from_pretrained(config_path)
+    vocab_size = tokenizer.vocab_size
+    (batch_size, block_size, max_iters, eval_interval, learning_rate, device,
+    eval_iters, n_embd, n_head, n_layer, dropout) = hyperparameters(config_path=config_path)
+
+elif model_name == "Shakespearean GPT":
+    config_path = './config/shakespearean_config.json'
+    data_path = './data/input.txt'
+    name = "Shakespearean GPT"
+    tokenizer: Tokenizer = Tokenizer()
+    tokenizer.from_pretrained(config_path)
+    vocab_size = tokenizer.vocab_size
+    (batch_size, block_size, max_iters, eval_interval, learning_rate, device,
+    eval_iters, n_embd, n_head, n_layer, dropout) = hyperparameters(config_path=config_path)
+    
+
+if model_name == "GPT":
+    input_text = st.text_area(
+        'Enter a prompt:', 'And then Ted said, "'
+    )
+else:
+    input_text = st.text_area(
+        'Enter a prompt:', 'Write a scene about ROMEO arguing with JULIET. \nROMEO:'
+    )
+
+temperature = st.sidebar.slider('Temperature:', 0.1, 1.0, 0.5, 0.1)
+max_tokens = st.sidebar.slider('Max Tokens:', 250, 1000, 500, 50)
+
+@st.cache_resource
+def load_model(path):
+    device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+    try:
+        model = GPTLanguageModel(
+            vocab_size, n_embd, block_size, n_head, n_layer, dropout, device, name=name
+        ).to(device)
+        state_dict = torch.load(
+            path, map_location=device)
+
+        model.load_state_dict(state_dict)
+        return model, device
+    except FileNotFoundError as e:
+        st.error(f"Don't forget to download the model weights from the link in the README.md file.")
+        return None, None
+
+
+model, device = load_model(path)
+
+
+if model:
+    if st.button('Generate Text'):
+        prompt = input_text
+        st.subheader(model.name)
+        input = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long, device=device)
+        generated_text = []
+        st.write(f":green[{prompt}]")
+        st.write_stream(decode_text(input, model, max_tokens, temperature))

config/config.json

@@ -0,0 +1,287 @@
+{
+  "vocab_size": 135,
+  "batch_size": 32,
+  "block_size": 256,
+  "max_iters": 3000,
+  "eval_interval": 300,
+  "learning_rate": 3e-5,
+  "eval_iters": 50,
+  "n_embd": 1024,
+  "n_head": 8,
+  "n_layer": 8,
+  "dropout": 0.3,
+  "encode": {
+    "\n": 0,
+    " ": 1,
+    "!": 2,
+    "\"": 3,
+    "#": 4,
+    "$": 5,
+    "%": 6,
+    "&": 7,
+    "'": 8,
+    "(": 9,
+    ")": 10,
+    "*": 11,
+    "+": 12,
+    ",": 13,
+    "-": 14,
+    ".": 15,
+    "/": 16,
+    "0": 17,
+    "1": 18,
+    "2": 19,
+    "3": 20,
+    "4": 21,
+    "5": 22,
+    "6": 23,
+    "7": 24,
+    "8": 25,
+    "9": 26,
+    ":": 27,
+    ";": 28,
+    "<": 29,
+    "=": 30,
+    ">": 31,
+    "?": 32,
+    "@": 33,
+    "A": 34,
+    "B": 35,
+    "C": 36,
+    "D": 37,
+    "E": 38,
+    "F": 39,
+    "G": 40,
+    "H": 41,
+    "I": 42,
+    "J": 43,
+    "K": 44,
+    "L": 45,
+    "M": 46,
+    "N": 47,
+    "O": 48,
+    "P": 49,
+    "Q": 50,
+    "R": 51,
+    "S": 52,
+    "T": 53,
+    "U": 54,
+    "V": 55,
+    "W": 56,
+    "X": 57,
+    "Y": 58,
+    "Z": 59,
+    "[": 60,
+    "\\": 61,
+    "]": 62,
+    "^": 63,
+    "_": 64,
+    "`": 65,
+    "a": 66,
+    "b": 67,
+    "c": 68,
+    "d": 69,
+    "e": 70,
+    "f": 71,
+    "g": 72,
+    "h": 73,
+    "i": 74,
+    "j": 75,
+    "k": 76,
+    "l": 77,
+    "m": 78,
+    "n": 79,
+    "o": 80,
+    "p": 81,
+    "q": 82,
+    "r": 83,
+    "s": 84,
+    "t": 85,
+    "u": 86,
+    "v": 87,
+    "w": 88,
+    "x": 89,
+    "y": 90,
+    "z": 91,
+    "{": 92,
+    "|": 93,
+    "}": 94,
+    "\u00a0": 95,
+    "\u00a3": 96,
+    "\u00b0": 97,
+    "\u00b2": 98,
+    "\u00b3": 99,
+    "\u00bc": 100,
+    "\u00bd": 101,
+    "\u00be": 102,
+    "\u00c6": 103,
+    "\u00c7": 104,
+    "\u00c8": 105,
+    "\u00c9": 106,
+    "\u00d7": 107,
+    "\u00dc": 108,
+    "\u00e0": 109,
+    "\u00e1": 110,
+    "\u00e2": 111,
+    "\u00e6": 112,
+    "\u00e7": 113,
+    "\u00e8": 114,
+    "\u00e9": 115,
+    "\u00ea": 116,
+    "\u00eb": 117,
+    "\u00ee": 118,
+    "\u00ef": 119,
+    "\u00f1": 120,
+    "\u00f2": 121,
+    "\u00f4": 122,
+    "\u00f6": 123,
+    "\u00f7": 124,
+    "\u00f9": 125,
+    "\u00fb": 126,
+    "\u00fc": 127,
+    "\u2013": 128,
+    "\u2014": 129,
+    "\u2018": 130,
+    "\u2019": 131,
+    "\u201c": 132,
+    "\u201d": 133,
+    "\ufeff": 134
+  },
+  "decode": {
+    "0": "\n",
+    "1": " ",
+    "2": "!",
+    "3": "\"",
+    "4": "#",
+    "5": "$",
+    "6": "%",
+    "7": "&",
+    "8": "'",
+    "9": "(",
+    "10": ")",
+    "11": "*",
+    "12": "+",
+    "13": ",",
+    "14": "-",
+    "15": ".",
+    "16": "/",
+    "17": "0",
+    "18": "1",
+    "19": "2",
+    "20": "3",
+    "21": "4",
+    "22": "5",
+    "23": "6",
+    "24": "7",
+    "25": "8",
+    "26": "9",
+    "27": ":",
+    "28": ";",
+    "29": "<",
+    "30": "=",
+    "31": ">",
+    "32": "?",
+    "33": "@",
+    "34": "A",
+    "35": "B",
+    "36": "C",
+    "37": "D",
+    "38": "E",
+    "39": "F",
+    "40": "G",
+    "41": "H",
+    "42": "I",
+    "43": "J",
+    "44": "K",
+    "45": "L",
+    "46": "M",
+    "47": "N",
+    "48": "O",
+    "49": "P",
+    "50": "Q",
+    "51": "R",
+    "52": "S",
+    "53": "T",
+    "54": "U",
+    "55": "V",
+    "56": "W",
+    "57": "X",
+    "58": "Y",
+    "59": "Z",
+    "60": "[",
+    "61": "\\",
+    "62": "]",
+    "63": "^",
+    "64": "_",
+    "65": "`",
+    "66": "a",
+    "67": "b",
+    "68": "c",
+    "69": "d",
+    "70": "e",
+    "71": "f",
+    "72": "g",
+    "73": "h",
+    "74": "i",
+    "75": "j",
+    "76": "k",
+    "77": "l",
+    "78": "m",
+    "79": "n",
+    "80": "o",
+    "81": "p",
+    "82": "q",
+    "83": "r",
+    "84": "s",
+    "85": "t",
+    "86": "u",
+    "87": "v",
+    "88": "w",
+    "89": "x",
+    "90": "y",
+    "91": "z",
+    "92": "{",
+    "93": "|",
+    "94": "}",
+    "95": "\u00a0",
+    "96": "\u00a3",
+    "97": "\u00b0",
+    "98": "\u00b2",
+    "99": "\u00b3",
+    "100": "\u00bc",
+    "101": "\u00bd",
+    "102": "\u00be",
+    "103": "\u00c6",
+    "104": "\u00c7",
+    "105": "\u00c8",
+    "106": "\u00c9",
+    "107": "\u00d7",
+    "108": "\u00dc",
+    "109": "\u00e0",
+    "110": "\u00e1",
+    "111": "\u00e2",
+    "112": "\u00e6",
+    "113": "\u00e7",
+    "114": "\u00e8",
+    "115": "\u00e9",
+    "116": "\u00ea",
+    "117": "\u00eb",
+    "118": "\u00ee",
+    "119": "\u00ef",
+    "120": "\u00f1",
+    "121": "\u00f2",
+    "122": "\u00f4",
+    "123": "\u00f6",
+    "124": "\u00f7",
+    "125": "\u00f9",
+    "126": "\u00fb",
+    "127": "\u00fc",
+    "128": "\u2013",
+    "129": "\u2014",
+    "130": "\u2018",
+    "131": "\u2019",
+    "132": "\u201c",
+    "133": "\u201d",
+    "134": "\ufeff"
+  }
+}

config/example-config.json

@@ -0,0 +1,12 @@
+{
+    "batch_size": 32,
+    "block_size": 256,
+    "max_iters": 3000,
+    "eval_interval": 300,
+    "learning_rate": 3e-5,
+    "eval_iters": 50,
+    "n_embd": 1024,
+    "n_head": 12,
+    "n_layer": 18,
+    "dropout": 0.3
+}

config/harpoon_config.json

@@ -0,0 +1,287 @@
+{
+  "vocab_size": 135,
+  "batch_size": 32,
+  "block_size": 256,
+  "max_iters": 3000,
+  "eval_interval": 300,
+  "learning_rate": 3e-5,
+  "eval_iters": 50,
+  "n_embd": 1024,
+  "n_head": 12,
+  "n_layer": 18,
+  "dropout": 0.3,
+  "encode": {
+    "\n": 0,
+    " ": 1,
+    "!": 2,
+    "\"": 3,
+    "#": 4,
+    "$": 5,
+    "%": 6,
+    "&": 7,
+    "'": 8,
+    "(": 9,
+    ")": 10,
+    "*": 11,
+    "+": 12,
+    ",": 13,
+    "-": 14,
+    ".": 15,
+    "/": 16,
+    "0": 17,
+    "1": 18,
+    "2": 19,
+    "3": 20,
+    "4": 21,
+    "5": 22,
+    "6": 23,
+    "7": 24,
+    "8": 25,
+    "9": 26,
+    ":": 27,
+    ";": 28,
+    "<": 29,
+    "=": 30,
+    ">": 31,
+    "?": 32,
+    "@": 33,
+    "A": 34,
+    "B": 35,
+    "C": 36,
+    "D": 37,
+    "E": 38,
+    "F": 39,
+    "G": 40,
+    "H": 41,
+    "I": 42,
+    "J": 43,
+    "K": 44,
+    "L": 45,
+    "M": 46,
+    "N": 47,
+    "O": 48,
+    "P": 49,
+    "Q": 50,
+    "R": 51,
+    "S": 52,
+    "T": 53,
+    "U": 54,
+    "V": 55,
+    "W": 56,
+    "X": 57,
+    "Y": 58,
+    "Z": 59,
+    "[": 60,
+    "\\": 61,
+    "]": 62,
+    "^": 63,
+    "_": 64,
+    "`": 65,
+    "a": 66,
+    "b": 67,
+    "c": 68,
+    "d": 69,
+    "e": 70,
+    "f": 71,
+    "g": 72,
+    "h": 73,
+    "i": 74,
+    "j": 75,
+    "k": 76,
+    "l": 77,
+    "m": 78,
+    "n": 79,
+    "o": 80,
+    "p": 81,
+    "q": 82,
+    "r": 83,
+    "s": 84,
+    "t": 85,
+    "u": 86,
+    "v": 87,
+    "w": 88,
+    "x": 89,
+    "y": 90,
+    "z": 91,
+    "{": 92,
+    "|": 93,
+    "}": 94,
+    "\u00a0": 95,
+    "\u00a3": 96,
+    "\u00b0": 97,
+    "\u00b2": 98,
+    "\u00b3": 99,
+    "\u00bc": 100,
+    "\u00bd": 101,
+    "\u00be": 102,
+    "\u00c6": 103,
+    "\u00c7": 104,
+    "\u00c8": 105,
+    "\u00c9": 106,
+    "\u00d7": 107,
+    "\u00dc": 108,
+    "\u00e0": 109,
+    "\u00e1": 110,
+    "\u00e2": 111,
+    "\u00e6": 112,
+    "\u00e7": 113,
+    "\u00e8": 114,
+    "\u00e9": 115,
+    "\u00ea": 116,
+    "\u00eb": 117,
+    "\u00ee": 118,
+    "\u00ef": 119,
+    "\u00f1": 120,
+    "\u00f2": 121,
+    "\u00f4": 122,
+    "\u00f6": 123,
+    "\u00f7": 124,
+    "\u00f9": 125,
+    "\u00fb": 126,
+    "\u00fc": 127,
+    "\u2013": 128,
+    "\u2014": 129,
+    "\u2018": 130,
+    "\u2019": 131,
+    "\u201c": 132,
+    "\u201d": 133,
+    "\ufeff": 134
+  },
+  "decode": {
+    "0": "\n",
+    "1": " ",
+    "2": "!",
+    "3": "\"",
+    "4": "#",
+    "5": "$",
+    "6": "%",
+    "7": "&",
+    "8": "'",
+    "9": "(",
+    "10": ")",
+    "11": "*",
+    "12": "+",
+    "13": ",",
+    "14": "-",
+    "15": ".",
+    "16": "/",
+    "17": "0",
+    "18": "1",
+    "19": "2",
+    "20": "3",
+    "21": "4",
+    "22": "5",
+    "23": "6",
+    "24": "7",
+    "25": "8",
+    "26": "9",
+    "27": ":",
+    "28": ";",
+    "29": "<",
+    "30": "=",
+    "31": ">",
+    "32": "?",
+    "33": "@",
+    "34": "A",
+    "35": "B",
+    "36": "C",
+    "37": "D",
+    "38": "E",
+    "39": "F",
+    "40": "G",
+    "41": "H",
+    "42": "I",
+    "43": "J",
+    "44": "K",
+    "45": "L",
+    "46": "M",
+    "47": "N",
+    "48": "O",
+    "49": "P",
+    "50": "Q",
+    "51": "R",
+    "52": "S",
+    "53": "T",
+    "54": "U",
+    "55": "V",
+    "56": "W",
+    "57": "X",
+    "58": "Y",
+    "59": "Z",
+    "60": "[",
+    "61": "\\",
+    "62": "]",
+    "63": "^",
+    "64": "_",
+    "65": "`",
+    "66": "a",
+    "67": "b",
+    "68": "c",
+    "69": "d",
+    "70": "e",
+    "71": "f",
+    "72": "g",
+    "73": "h",
+    "74": "i",
+    "75": "j",
+    "76": "k",
+    "77": "l",
+    "78": "m",
+    "79": "n",
+    "80": "o",
+    "81": "p",
+    "82": "q",
+    "83": "r",
+    "84": "s",
+    "85": "t",
+    "86": "u",
+    "87": "v",
+    "88": "w",
+    "89": "x",
+    "90": "y",
+    "91": "z",
+    "92": "{",
+    "93": "|",
+    "94": "}",
+    "95": "\u00a0",
+    "96": "\u00a3",
+    "97": "\u00b0",
+    "98": "\u00b2",
+    "99": "\u00b3",
+    "100": "\u00bc",
+    "101": "\u00bd",
+    "102": "\u00be",
+    "103": "\u00c6",
+    "104": "\u00c7",
+    "105": "\u00c8",
+    "106": "\u00c9",
+    "107": "\u00d7",
+    "108": "\u00dc",
+    "109": "\u00e0",
+    "110": "\u00e1",
+    "111": "\u00e2",
+    "112": "\u00e6",
+    "113": "\u00e7",
+    "114": "\u00e8",
+    "115": "\u00e9",
+    "116": "\u00ea",
+    "117": "\u00eb",
+    "118": "\u00ee",
+    "119": "\u00ef",
+    "120": "\u00f1",
+    "121": "\u00f2",
+    "122": "\u00f4",
+    "123": "\u00f6",
+    "124": "\u00f7",
+    "125": "\u00f9",
+    "126": "\u00fb",
+    "127": "\u00fc",
+    "128": "\u2013",
+    "129": "\u2014",
+    "130": "\u2018",
+    "131": "\u2019",
+    "132": "\u201c",
+    "133": "\u201d",
+    "134": "\ufeff"
+  }
+}

config/shakespearean_config.json

@@ -0,0 +1,147 @@
+{
+  "vocab_size": 65,
+  "batch_size": 32,
+  "block_size": 256,
+  "max_iters": 3000,
+  "eval_interval": 300,
+  "learning_rate": 3e-5,
+  "eval_iters": 50,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 6,
+  "dropout": 0.3,
+  "encode": {
+    "\n": 0,
+    " ": 1,
+    "!": 2,
+    "$": 3,
+    "&": 4,
+    "'": 5,
+    ",": 6,
+    "-": 7,
+    ".": 8,
+    "3": 9,
+    ":": 10,
+    ";": 11,
+    "?": 12,
+    "A": 13,
+    "B": 14,
+    "C": 15,
+    "D": 16,
+    "E": 17,
+    "F": 18,
+    "G": 19,
+    "H": 20,
+    "I": 21,
+    "J": 22,
+    "K": 23,
+    "L": 24,
+    "M": 25,
+    "N": 26,
+    "O": 27,
+    "P": 28,
+    "Q": 29,
+    "R": 30,
+    "S": 31,
+    "T": 32,
+    "U": 33,
+    "V": 34,
+    "W": 35,
+    "X": 36,
+    "Y": 37,
+    "Z": 38,
+    "a": 39,
+    "b": 40,
+    "c": 41,
+    "d": 42,
+    "e": 43,
+    "f": 44,
+    "g": 45,
+    "h": 46,
+    "i": 47,
+    "j": 48,
+    "k": 49,
+    "l": 50,
+    "m": 51,
+    "n": 52,
+    "o": 53,
+    "p": 54,
+    "q": 55,
+    "r": 56,
+    "s": 57,
+    "t": 58,
+    "u": 59,
+    "v": 60,
+    "w": 61,
+    "x": 62,
+    "y": 63,
+    "z": 64
+  },
+  "decode": {
+    "0": "\n",
+    "1": " ",
+    "2": "!",
+    "3": "$",
+    "4": "&",
+    "5": "'",
+    "6": ",",
+    "7": "-",
+    "8": ".",
+    "9": "3",
+    "10": ":",
+    "11": ";",
+    "12": "?",
+    "13": "A",
+    "14": "B",
+    "15": "C",
+    "16": "D",
+    "17": "E",
+    "18": "F",
+    "19": "G",
+    "20": "H",
+    "21": "I",
+    "22": "J",
+    "23": "K",
+    "24": "L",
+    "25": "M",
+    "26": "N",
+    "27": "O",
+    "28": "P",
+    "29": "Q",
+    "30": "R",
+    "31": "S",
+    "32": "T",
+    "33": "U",
+    "34": "V",
+    "35": "W",
+    "36": "X",
+    "37": "Y",
+    "38": "Z",
+    "39": "a",
+    "40": "b",
+    "41": "c",
+    "42": "d",
+    "43": "e",
+    "44": "f",
+    "45": "g",
+    "46": "h",
+    "47": "i",
+    "48": "j",
+    "49": "k",
+    "50": "l",
+    "51": "m",
+    "52": "n",
+    "53": "o",
+    "54": "p",
+    "55": "q",
+    "56": "r",
+    "57": "s",
+    "58": "t",
+    "59": "u",
+    "60": "v",
+    "61": "w",
+    "62": "x",
+    "63": "y",
+    "64": "z"
+  }
+}

core/layers/__init__.py

@@ -0,0 +1 @@
+from .layers import Block, FeedForward, MultiHeadAttention, Head, RoPE, LlamaBlock, RMSNorm

core/layers/layers.py

@@ -0,0 +1,207 @@
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+import tqdm
+
+
+class Head(nn.Module):
+    """One head of self-attention."""
+
+    def __init__(self, n_embd, head_size, block_size, dropout):
+        super().__init__()
+        self.key = nn.Linear(n_embd, head_size, bias=False)
+        self.query = nn.Linear(n_embd, head_size, bias=False)
+        self.value = nn.Linear(n_embd, head_size, bias=False)
+        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x):
+        B, T, C = x.shape
+        k = self.key(x)
+        q = self.query(x)
+        wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5
+        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
+        wei = F.softmax(wei, dim=-1)
+        wei = self.dropout(wei)
+        v = self.value(x)
+        out = wei @ v
+        return out
+
+class MultiHeadAttention(nn.Module):
+    """Multiple heads of self-attention in parallel."""
+
+    def __init__(self, n_embd, n_head, block_size, dropout):
+        super().__init__()
+        assert n_embd % n_head == 0, f"n_embd ({n_embd}) must be divisible by num_heads ({n_head})"
+
+        self.n_embd = n_embd
+        self.n_head = n_head
+        self.head_size = n_embd // n_head
+
+        self.heads = nn.ModuleList([Head(n_embd, self.head_size, block_size, dropout) for _ in range(n_head)])
+        self.proj = nn.Linear(n_embd, n_embd)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x):
+        out = torch.cat([h(x) for h in self.heads], dim=-1)
+        out = self.dropout(self.proj(out))
+        return out
+
+class FeedForward(nn.Module):
+    """A simple linear layer followed by a non-linearity."""
+
+    def __init__(self, n_embd, dropout):
+        super().__init__()
+        self.net = nn.Sequential(
+            nn.Linear(n_embd, 4 * n_embd),
+            nn.ReLU(),
+            nn.Linear(4 * n_embd, n_embd),
+            nn.Dropout(dropout),
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+class Block(nn.Module):
+    """Transformer block: communication followed by computation."""
+
+    def __init__(self, n_embd, n_head, block_size, dropout):
+        super().__init__()
+        self.sa = MultiHeadAttention(n_embd, n_head, block_size, dropout)
+        self.ffwd = FeedForward(n_embd, dropout)
+        self.ln1 = nn.LayerNorm(n_embd)
+        self.ln2 = nn.LayerNorm(n_embd)
+
+    def forward(self, x):
+        x = x + self.sa(self.ln1(x))
+        x = x + self.ffwd(self.ln2(x))
+        return x
+    
+
+class RoPE(nn.Module):
+    """Rotary Positional Encoding (RoPE) layer."""
+
+    def __init__(self, embd_dim, max_freq=10):
+        super().__init__()
+        self.embd_dim = embd_dim
+        self.max_freq = max_freq
+        self.freqs = 2 ** torch.linspace(0, max_freq - 1, embd_dim // 2) * torch.pi
+        self.inv_freqs = 1. / self.freqs
+
+    def forward(self, x):
+        x = x + torch.sin(x @ self.freqs) * self.inv_freqs
+        x = x + torch.cos(x @ self.freqs) * self.inv_freqs
+        return x
+
+
+class RMSNorm(nn.Module):
+    """Root Mean Square Layer Normalization (RMSNorm)."""
+
+    def __init__(self, embd_dim, epsilon=1e-8):
+        super().__init__()
+        self.embd_dim = embd_dim
+        self.epsilon = epsilon
+        self.gamma = nn.Parameter(torch.ones(embd_dim))
+        self.beta = nn.Parameter(torch.zeros(embd_dim))
+
+    def forward(self, x: torch.Tensor):
+        mean = x.mean(-1, keepdim=True)
+        variance = x.var(-1, keepdim=True)
+        x = x - mean
+        x = x / torch.sqrt(variance + self.epsilon)
+        x = x * self.gamma + self.beta
+        return x
+    
+
+class LlamaFFN(nn.Module):
+    """Feed-forward network of the LLAMA model with SwiGLU activation."""
+
+    def __init__(self, n_embd, dropout):
+        super().__init__()
+        self.linear1 = nn.Linear(n_embd, 4 * n_embd)
+        self.linear2 = nn.Linear(4 * n_embd, n_embd)
+        self.dropout = nn.Dropout(dropout)
+
+    def swiglu(self, x):
+        """Applies SwiGLU activation."""
+        x1, x2 = torch.chunk(x, 2, dim=-1)
+        return x1 * F.silu(x2)
+
+    def forward(self, x):
+        x = self.linear1(x)
+        x = self.swiglu(x)
+        x = self.dropout(x)
+        x = self.linear2(x)
+        return x
+
+
+class AttentionHeadWithKVCacheAndRoPE(nn.Module):
+    """One head of self-attention with key and value cache and RoPE."""
+
+    def __init__(self, n_embd, head_size, block_size, dropout):
+        super().__init__()
+        self.key = nn.Linear(n_embd, head_size, bias=False)
+        self.query = nn.Linear(n_embd, head_size, bias=False)
+        self.value = nn.Linear(n_embd, head_size, bias=False)
+        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))
+        self.dropout = nn.Dropout(dropout)
+        self.pe = RoPE(head_size)
+        self.ln = RMSNorm(n_embd)
+
+    def forward(self, x, kv_cache):
+        B, T, C = x.shape
+        k = self.key(x)
+        q = self.query(x)
+        v = self.value(x)
+        if kv_cache is not None:
+            k = torch.cat([kv_cache['k'], k], dim=1)
+            v = torch.cat([kv_cache['v'], v], dim=1)
+        wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5
+        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
+        wei = F.softmax(wei, dim=-1)
+        wei = self.dropout(wei)
+        out = wei @ v
+        if kv_cache is None:
+            kv_cache = {'k': k, 'q': q, 'v': v}
+        else:
+            kv_cache['k'] = k
+            kv_cache['q'] = q
+            kv_cache['v'] = v
+        return self.pe(out) + x
+
+
+class MultiHeadAttentionWithKVCacheAndRoPE(nn.Module):
+    """Multiple heads of self-attention in parallel."""
+
+    def __init__(self, n_embd, n_head, block_size, dropout):
+        super().__init__()
+        assert n_embd % n_head == 0, f"n_embd ({n_embd}) must be divisible by num_heads ({n_head})"
+
+        self.n_embd = n_embd
+        self.n_head = n_head
+        self.head_size = n_embd // n_head
+
+        self.heads = nn.ModuleList([AttentionHeadWithKVCacheAndRoPE(n_embd, self.head_size, block_size, dropout) for _ in range(n_head)])
+        self.proj = nn.Linear(n_embd, n_embd)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x, kv_cache):
+        out = torch.cat([h(x, kv_cache) for h in self.heads], dim=-1)
+        out = self.dropout(self.proj(out))
+        return out
+
+
+class LlamaBlock(nn.Module):
+    """LLAMA block: communication followed by computation."""
+
+    def __init__(self, n_embd, n_head, block_size, dropout):
+        super().__init__()
+        self.ln1 = RMSNorm(n_embd)
+        self.sa = MultiHeadAttentionWithKVCacheAndRoPE(n_embd, n_head, block_size, dropout)
+        self.ln2 = RMSNorm(n_embd)
+        self.ffwd = LlamaFFN(n_embd, dropout)
+    
+    def forward(self, x, kv_cache):
+        x = x + self.sa(self.ln1(x), kv_cache)
+        x = x + self.ffwd(self.ln2(x))
+        return x

core/models/__init__.py

@@ -0,0 +1 @@
+from . import gpt

core/models/gpt.py

@@ -0,0 +1,64 @@
+import torch
+import torch.nn as nn
+import tqdm
+from torch.nn import functional as F
+from  core.layers import Block
+
+class GPTLanguageModel(nn.Module):
+
+    def __init__(self, vocab_size, n_embd, block_size, n_head, n_layer, dropout, device, name = "GPT"):
+        super().__init__()
+        self.name = name
+        self.block_size = block_size
+        self.device = device
+        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
+        self.position_embedding_table = nn.Embedding(block_size, n_embd)
+        self.blocks = nn.Sequential(*[Block(n_embd, n_head, block_size, dropout) for _ in range(n_layer)])
+        self.ln_f = nn.LayerNorm(n_embd)
+        self.lm_head = nn.Linear(n_embd, vocab_size)
+        self.apply(self._init_weights)
+        self.history = {}
+        self.vocab_size = vocab_size
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+            if module.bias is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+    def forward(self, idx, targets=None):
+        B, T = idx.shape
+
+        assert torch.all(idx < self.vocab_size), f"Input indices must be less than vocab_size ({self.vocab_size})"
+        assert T <= self.block_size, f"Input sequence length ({T}) must be <= block_size ({self.block_size})"
+
+        tok_emb = self.token_embedding_table(idx)
+        pos_emb = self.position_embedding_table(torch.arange(T, device=idx.device))
+        x = tok_emb + pos_emb
+        x = self.blocks(x)
+        x = self.ln_f(x)
+        logits = self.lm_head(x)
+
+        if targets is None:
+            loss = None
+        else:
+            B, T, C = logits.shape
+            logits = logits.view(B * T, C)
+            targets = targets.view(B * T)
+            loss = F.cross_entropy(logits, targets)
+
+        return logits, loss
+
+    def generate(self, idx, max_new_tokens, max_seq_length=200, temperature=1.0):
+        for _ in range(max_new_tokens):
+            if idx.size(1) > max_seq_length:
+                idx = idx[:, -max_seq_length:]
+            idx_cond = idx[:, -self.block_size:]
+            logits, _ = self(idx_cond)
+            logits = logits[:, -1, :] / temperature
+            probs = F.softmax(logits, dim=-1)
+            idx_next = torch.multinomial(probs, num_samples=1)
+            idx = torch.cat((idx, idx_next), dim=1)
+            yield idx

core/models/llama.py

@@ -0,0 +1,51 @@
+import torch
+import torch.nn as nn
+import tqdm
+from torch.nn import functional as F
+from  core.layers import LlamaBlock, RMSNorm
+
+class LlamaLanguageModel(nn.Module):
+
+    def __init__(self, vocab_size, n_embd, block_size, n_head, n_layer, dropout, device, name = "llama"):
+        super().__init__()
+        self.name = name
+        self.block_size = block_size
+        self.device = device
+        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
+        self.blocks = nn.Sequential(*[LlamaBlock(n_embd, n_head, block_size, dropout) for _ in range(n_layer)])
+        self.ln_f = RMSNorm(n_embd)
+        self.lm_head = nn.Linear(n_embd, vocab_size)
+        self.apply(self._init_weights)
+        self.history = {}
+        self.vocab_size = vocab_size
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+            if module.bias is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+    def forward(self, idx):
+        B, T = idx.shape
+        kv_cache = None
+        token_embeddings = self.token_embedding_table(idx)
+        for block in self.blocks:
+            token_embeddings = block(token_embeddings, kv_cache)
+        token_embeddings = self.ln_f(token_embeddings)
+        logits = self.lm_head(token_embeddings)
+        return logits, token_embeddings
+
+
+    def generate(self, idx, max_new_tokens, max_seq_length=200, temperature=1.0):
+        for _ in range(max_new_tokens):
+            if idx.size(1) > max_seq_length:
+                idx = idx[:, -max_seq_length:]
+            idx_cond = idx[:, -self.block_size:]
+            logits, _ = self(idx_cond)
+            logits = logits[:, -1, :] / temperature
+            probs = F.softmax(logits, dim=-1)
+            idx_next = torch.multinomial(probs, num_samples=1)
+            idx = torch.cat((idx, idx_next), dim=1)
+            yield idx

core/tokenizers/__init__.py

@@ -0,0 +1 @@
+from . import tokenizer

core/tokenizers/tokenizer.py

@@ -0,0 +1,115 @@
+import json
+import os
+from typing import Iterable
+import torch
+
+class Tokenizer:
+    def __init__(self, data_path: str = None):
+        self.config = None
+        self.stoi = None
+        self.itos = None
+        self.vocab_size = None
+        if data_path:
+            self.data = self.load_data(data_path)
+        else:
+            self.data = None
+    
+    def from_pretrained(self, config_path: str):
+        with open(config_path) as f:
+            config = json.load(f)
+        self.config = config
+        if 'encode' not in config:
+            raise ValueError("Config file must contain an 'encode' key.")
+        if 'decode' not in config:
+            raise ValueError("Config file must contain a 'decode' key.")
+        if 'vocab_size' not in config:
+            raise ValueError("Config file must contain a 'vocab_size' key.")
+        stoi = config['encode']
+        self.stoi = {k: int(v) for k, v in stoi.items()}
+        itos = config['decode']
+        self.itos = {int(k): v for k, v in itos.items()}
+        self.vocab_size = config['vocab_size']
+        return self
+    
+    def load_data(self, path: str) -> str:
+        if not os.path.exists(path):
+            raise FileNotFoundError("File not found.")
+        if not path.endswith('.txt'):
+            raise ValueError("File must be a text file.")
+        with open(path, 'r', encoding='utf-8') as f:
+            text = f.read()
+        chars = sorted(list(set(text)))
+        vocab_size = len(chars)
+        stoi = {ch: i for i, ch in enumerate(chars)}
+        itos = {i: ch for i, ch in enumerate(chars)}
+        self.config = {"vocab_size": vocab_size, "encode": stoi, "decode": itos}
+        self.stoi = stoi
+        self.itos = itos
+        data = torch.tensor(self(text), dtype=torch.long)
+        n = int(0.9*len(data))
+        train_data = data[:n]
+        val_data = data[n:]
+        self.train_data = train_data
+        self.val_data = val_data
+        self.vocab_size = vocab_size
+        return text
+
+    def __repr__(self) -> str:
+        if self.config:
+            return f"Tokenizer(config={self.config})"
+        else:
+            return f"Tokenizer()"
+    
+    def __str__(self) -> str:
+        if self.config:
+            return f"Tokenizer(config_path={self.config})"
+        else:
+            return f"Tokenizer()"
+    
+    def __len__(self) -> int:
+        return len(self.stoi)
+    
+    def __getitem__(self, key: str) -> int:
+        return self.stoi[key]
+    
+    def __contains__(self, key: str) -> bool:
+        return key in self.stoi
+    
+    def __iter__(self):
+        return iter(self.stoi)
+    
+    def __reversed__(self):
+        return reversed(self.stoi)
+    
+    def keys(self):
+        return self.stoi.keys()
+    
+    def values(self):
+        return self.stoi.values()
+    
+    def items(self):
+        return self.stoi.items()
+    
+    def __call__(self, *args, **kwds) -> list[int]:
+        return self.encode(*args, **kwds)
+
+    def encode(self, s: str | list[str]) -> list[int]:
+        if isinstance(s, str):
+            return [self.stoi[c] for c in s]
+        elif isinstance(s, list):
+            return [[self.stoi[i] for i in c] for c in s]
+        else:
+            raise ValueError("Input must be a string or a list of strings.")
+
+    def decode(self, l: list[int]) -> str:
+        if isinstance(l[0], int):
+            return ''.join([self.itos[i] for i in l])
+        elif isinstance(l[0], Iterable):
+            return [''.join([self.itos[i] for i in c]) for c in l]
+        else:
+            raise ValueError("Input must be a list of integers or a list of list of integers.")
+    
+    def save_pretrained(self, path: str) -> str:
+        with open(path + 'vocab.json', 'w') as f:
+            json.dump(self.config, f)
+        return "Tokenizer saved at {}.".format(path)

core/utils/__init__.py

@@ -0,0 +1 @@
+from . import gptutils, preprocessing

core/utils/gptutils.py

@@ -0,0 +1,71 @@
+import torch
+import json
+
+# ------------ Hyperparameters ------------
+def hyperparameters(config_path: str):
+    with open(config_path) as f:
+        config = json.load(f)
+
+    batch_size = config['batch_size']
+    block_size = config['block_size']
+    max_iters = config['max_iters']
+    eval_interval = config['eval_interval']
+    learning_rate = config['learning_rate']
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    eval_iters = config['eval_iters']
+    n_embd = config['n_embd']
+    n_head = config['n_head']
+    n_layer = config['n_layer']
+    dropout = config['dropout']
+    return (batch_size, block_size, max_iters, eval_interval, learning_rate,
+            device, eval_iters, n_embd, n_head, n_layer, dropout)
+# ----------------------------------------
+
+def load_data(path) -> tuple[torch.Tensor, torch.Tensor, int, callable, callable]:
+    with open(path, 'r', encoding='utf-8') as f:
+        text = f.read()
+
+    # words = text.split()
+    # vocab_size = len(words)
+    # stoi = {word: i for i, word in enumerate(words)}
+    # itos = {i: word for i, word in enumerate(words)}
+    # def encode(s): return [stoi[w] for w in s.split()]
+    # def decode(ids): return ' '.join([itos[i] for i in ids])
+
+    chars = sorted(list(set(text)))
+    vocab_size = len(chars)
+    stoi = {ch: i for i, ch in enumerate(chars)}
+    itos = {i: ch for i, ch in enumerate(chars)}
+    def encode(s): return [stoi[c] for c in s]
+    def decode(l): return ''.join([itos[i] for i in l])
+    data = torch.tensor(encode(text), dtype=torch.long)
+    n = int(0.9*len(data))
+    train_data = data[:n]
+    val_data = data[n:]
+    
+
+    return train_data, val_data, vocab_size, encode, decode
+
+
+def get_batch(split, train_data, val_data, device, block_size, batch_size):
+    data = train_data if split == 'train' else val_data
+    ix = torch.randint(len(data) - block_size, (batch_size,))
+    x = torch.stack([data[i:i+block_size] for i in ix])
+    y = torch.stack([data[i+1:i+block_size+1] for i in ix])
+    x, y = x.to(device), y.to(device)
+    return x, y
+
+
+@torch.no_grad()
+def estimate_loss(model, get_batch, eval_iters, train_data, val_data, device, block_size, batch_size):
+    out = {}
+    model.eval()
+    for split in ['train', 'val']:
+        losses = torch.zeros(eval_iters)
+        for k in range(eval_iters):
+            X, Y = get_batch(split, train_data, val_data, device, block_size, batch_size)
+            logits, loss = model(X, Y)
+            losses[k] = loss.item()
+        out[split] = losses.mean()
+    model.train()
+    return out

core/utils/preprocessing.py

@@ -0,0 +1,75 @@
+import numpy as np
+import torch
+
+
+class DataLoader:
+    def __init__(self, data_path):
+        self.data_path = data_path
+        self.batch_size = None
+        self.block_size = None
+        self.data = None
+        self.train_data = None
+        self.val_data = None
+
+    def load_data(self, block_size=128, split=0.8, batch_size=64, device='cpu'):
+        with open(self.data_path, 'r') as f:
+            data = f.read()
+        self.block_size = block_size
+        self.batch_size = batch_size
+        self.device = device
+        self.data = data
+
+    def __len__(self):
+        return int(np.ceil(len(self.data) / self.batch_size))
+
+    def __getitem__(self, index):
+        indexes = self.indexes[index *
+                               self.batch_size:(index + 1) * self.batch_size]
+        batch = [self.data[i] for i in indexes]
+        batch = np.array(batch)
+        return batch
+
+    def get_batch(self, split, device='cpu'):
+        if self.data is None:
+            raise ValueError('Data not loaded')
+        data = self.train_data if split == 'train' else self.val_data
+        ix = torch.randint(len(data) - self.block_size, (self.batch_size,))
+        x = torch.stack([data[i:i+self.block_size] for i in ix])
+        y = torch.stack([data[i+1:i+self.block_size+1] for i in ix])
+        x, y = x.to(device), y.to(device)
+        return x, y
+
+
+class Encoder:
+    def __init__(self, data, type='char'):
+        self.data = data
+        self.type = type
+        self.vocab_size = None
+        if type == 'char':
+            self.chars = sorted(list(set(data)))
+            self.stoi = {ch: i for i, ch in enumerate(self.chars)}
+            self.itos = {i: ch for i, ch in enumerate(self.chars)}
+            self.vocab_size = len(self.chars)
+        elif type == 'word':
+            self.words = data.split()
+            self.stoi = {word: i for i, word in enumerate(self.words)}
+            self.itos = {i: word for i, word in enumerate(self.words)}
+            self.vocab_size = len(self.words)
+        else:
+            raise ValueError('Type must be either "char" or "word"')
+
+    def encode(self, string: str):
+        if self.type == 'char':
+            return torch.tensor([self.stoi[c] for c in string])
+        elif self.type == 'word':
+            return torch.tensor([self.stoi[w] for w in string.split()])
+        else:
+            raise ValueError('Type must be either "char" or "word"')
+
+    def decode(self, ids: list):
+        if self.type == 'char':
+            return ''.join([self.itos[i] for i in ids])
+        elif self.type == 'word':
+            return ' '.join([self.itos[i] for i in ids])
+        else:
+            raise ValueError('Type must be either "char" or "word"')

requirements.txt

@@ -0,0 +1,96 @@
+altair==5.2.0
+annotated-types==0.6.0
+anyio==4.3.0
+asttokens==2.4.1
+attrs==23.2.0
+blinker==1.7.0
+cachetools==5.3.3
+certifi==2024.2.2
+charset-normalizer==3.3.2
+click==8.1.7
+colorama==0.4.6
+comm==0.2.2
+debugpy==1.8.1
+decorator==5.1.1
+dnspython==2.6.1
+email_validator==2.1.1
+executing==2.0.1
+fastapi==0.110.3
+filelock==3.13.1
+fsspec==2024.2.0
+gitdb==4.0.11
+GitPython==3.1.42
+h11==0.14.0
+httpcore==1.0.5
+httptools==0.6.1
+httpx==0.27.0
+idna==3.6
+ipykernel==6.29.4
+ipython==8.24.0
+itsdangerous==2.2.0
+jedi==0.19.1
+Jinja2==3.1.3
+joblib==1.4.0
+jsonschema==4.21.1
+jsonschema-specifications==2023.12.1
+jupyter_client==8.6.1
+jupyter_core==5.7.2
+markdown-it-py==3.0.0
+MarkupSafe==2.1.5
+matplotlib-inline==0.1.7
+mdurl==0.1.2
+mpmath==1.3.0
+nest-asyncio==1.6.0
+networkx==3.2.1
+numpy==1.26.4
+orjson==3.10.2
+packaging==23.2
+pandas==2.2.1
+parso==0.8.4
+pillow==10.2.0
+platformdirs==4.2.1
+prompt-toolkit==3.0.43
+protobuf==4.25.3
+psutil==5.9.8
+pure-eval==0.2.2
+pyarrow==15.0.2
+pydantic==2.7.1
+pydantic-extra-types==2.7.0
+pydantic-settings==2.2.1
+pydantic_core==2.18.2
+pydeck==0.8.1b0
+Pygments==2.17.2
+python-dateutil==2.9.0.post0
+python-dotenv==1.0.1
+python-multipart==0.0.9
+pytz==2024.1
+PyYAML==6.0.1
+pyzmq==26.0.2
+referencing==0.34.0
+regex==2024.4.28
+requests==2.31.0
+rich==13.7.1
+rpds-py==0.18.0
+six==1.16.0
+smmap==5.0.1
+sniffio==1.3.1
+stack-data==0.6.3
+starlette==0.37.2
+streamlit==1.32.2
+sympy==1.12
+tenacity==8.2.3
+toml==0.10.2
+toolz==0.12.1
+torch==2.2.1
+tornado==6.4
+tqdm==4.66.2
+traitlets==5.14.3
+typing_extensions==4.10.0
+tzdata==2024.1
+ujson==5.9.0
+urllib3==2.2.1
+uvicorn==0.29.0
+watchdog==4.0.0
+watchfiles==0.21.0
+wcwidth==0.2.13
+websockets==12.0