Upload tokenizer

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

special_tokens_map.json

@@ -0,0 +1 @@
+{}

tokenizer.py

@@ -0,0 +1,96 @@
+from typing import List, Optional, Dict, Tuple
+import json
+import os
+from transformers import PreTrainedTokenizer
+
+
+class AlphabetTokenizer(PreTrainedTokenizer):
+    vocab_files_names = {"vocab_file": "vocab.json"}
+    special_tokens_dict = {
+        'unk_token': '[UNK]',
+        'sep_token': '[SEP]',
+        'pad_token': '[PAD]',
+        'cls_token': '[CLS]',
+        'mask_token': '[MASK]'
+    }
+
+    def __init__(self, **kwargs):
+        self.alphabet = [chr(i) for i in range(65, 65+19)] + [chr(i).lower() for i in range(65, 65+19)] + [str(i) for i in range(0, 10)] + ['.', '+', '-', ' ', 'W']
+        self.vocab = {char: i for i, char in enumerate(self.alphabet)}
+        self.inv_vocab = {i: char for char, i in self.vocab.items()}
+
+        # Initialize with default special tokens
+        super().__init__(
+            **kwargs
+        )
+        # override default _add_tokens of special tokens, and we add manually afterwards
+        self._added_tokens_decoder = {}
+        self.add_special_tokens(self.special_tokens_dict)
+
+    def get_vocab(self) -> Dict[str, int]:
+        return dict(self.vocab)
+
+    def _tokenize(self, text: str) -> List[str]:
+        return [char for char in text if char in self.alphabet or char in self.vocab]
+
+    def _convert_token_to_id(self, token: str) -> int:
+        return self.vocab.get(token, self.vocab.get(self.unk_token))
+
+    def _convert_id_to_token(self, index: int) -> str:
+        return self.inv_vocab.get(index, self.unk_token)
+
+    def convert_tokens_to_string(self, tokens: List[str]) -> str:
+        return ''.join(tokens)
+
+    def build_inputs_with_special_tokens(self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None) -> List[int]:
+        if token_ids_1 is None:
+            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
+        cls = [self.cls_token_id]
+        sep = [self.sep_token_id]
+        return cls + token_ids_0 + sep + token_ids_1 + sep
+
+    def add_special_tokens(self, special_tokens_dict: Dict[str, str]) -> int:
+        """Override add_special_tokens to update both vocab and inv_vocab"""
+        added_tokens = 0
+        for token_name, token in special_tokens_dict.items():
+            if token not in self.vocab:
+                self.vocab[token] = len(self.vocab)
+                self.inv_vocab[len(self.inv_vocab)] = token
+                self.all_special_tokens_extended.append(token)
+                setattr(self, f"{token_name}_token", token)
+                added_tokens += 1
+        return added_tokens
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self.vocab)
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
+        """Save the vocabulary and special tokens file to a directory."""
+        if not os.path.isdir(save_directory):
+            raise ValueError(f"Vocabulary path ({save_directory}) should be a directory")
+
+        vocab_file = os.path.join(
+            save_directory, (filename_prefix + "-" if filename_prefix else "") + "vocab.json"
+        )
+
+        with open(vocab_file, "w", encoding="utf-8") as f:
+            f.write(json.dumps(self.vocab, ensure_ascii=False))
+
+        return (vocab_file,)
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs):
+        """Load the tokenizer from a pretrained model vocabulary."""
+        tokenizer = cls(*init_inputs, **kwargs)
+        vocab_file = os.path.join(pretrained_model_name_or_path, tokenizer.vocab_files_names["vocab_file"])
+        if os.path.isfile(vocab_file):
+            with open(vocab_file, "r", encoding="utf-8") as f:
+                vocab = json.load(f)
+            tokenizer.vocab = vocab
+            tokenizer.inv_vocab = {v: k for k, v in vocab.items()}
+
+        # override default _add_tokens of special tokens, and we added manually
+        tokenizer._added_tokens_decoder = {}
+        tokenizer.add_special_tokens(cls.special_tokens_dict)
+        return tokenizer

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "added_tokens_decoder": {},
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenizer.AlphabetTokenizer",
+      null
+    ]
+  },
+  "clean_up_tokenization_spaces": true,
+  "model_max_length": 2048,
+  "tokenizer_class": "AlphabetTokenizer"
+}

vocab.json

@@ -0,0 +1 @@
+{"A": 0, "B": 1, "C": 2, "D": 3, "E": 4, "F": 5, "G": 6, "H": 7, "I": 8, "J": 9, "K": 10, "L": 11, "M": 12, "N": 13, "O": 14, "P": 15, "Q": 16, "R": 17, "S": 18, "a": 19, "b": 20, "c": 21, "d": 22, "e": 23, "f": 24, "g": 25, "h": 26, "i": 27, "j": 28, "k": 29, "l": 30, "m": 31, "n": 32, "o": 33, "p": 34, "q": 35, "r": 36, "s": 37, "0": 38, "1": 39, "2": 40, "3": 41, "4": 42, "5": 43, "6": 44, "7": 45, "8": 46, "9": 47, ".": 48, "+": 49, "-": 50, " ": 51, "W": 52, ">": 53, "X": 54, "[UNK]": 55, "[SEP]": 56, "[PAD]": 57, "[CLS]": 58, "[MASK]": 59}