Update README.md

README.md

@@ -1,3 +1,139 @@
----
-license: mit
----
+---
+license: mit
+language:
+- sk
+datasets:
+- oscar-corpus/OSCAR-2109
+pipeline_tag: fill-mask
+library_name: transformers
+---
+# Slovak Morphological Baby Language Model (SK_Morph_BLM)
+
+**SK_Morph_BLM** is a pretrained small language model for the Slovak language, based on the RoBERTa architecture. The model utilizes a custom morphological tokenizer specifically designed for the Slovak language, which focuses on **preserving the integrity of root morphemes**. This tokenizer is not compatible with the standard `RobertaTokenizer` from the Hugging Face library due to its unique approach to tokenization. The model is case-insensitive, meaning it operates in lowercase. While the pretrained model can be used for masked language modeling, it is primarily intended for fine-tuning on downstream NLP tasks.
+
+## How to Use the Model
+
+To use the SK_Morph_BLM model, follow these steps:
+
+```python
+import torch
+import sys
+from transformers import AutoModelForMaskedLM
+from huggingface_hub import snapshot_download
+
+# Download the repository from Hugging Face and append the path to sys.path
+repo_path = snapshot_download(repo_id="daviddrzik/SK_Morph_BLM")
+sys.path.append(repo_path)
+
+# Import the custom tokenizer from the downloaded repository
+from SKMT_lib_v2.SKMT_BPE import SKMorfoTokenizer
+
+# Initialize the tokenizer and model
+tokenizer = SKMorfoTokenizer()
+model = AutoModelForMaskedLM.from_pretrained("daviddrzik/SK_Morph_BLM")
+
+# Function to fill in the masked token in a given text
+def fill_mask(tokenized_text, tokenizer, model, top_k=5):
+    inputs = tokenizer.tokenize(tokenized_text.lower(), max_length=256, return_tensors='pt', return_subword=False)
+    mask_token_index = torch.where(inputs["input_ids"][0] == 4)[0]
+    with torch.no_grad():
+        predictions = model(**inputs)
+
+    topk_tokens = torch.topk(predictions.logits[0, mask_token_index], k=top_k, dim=-1).indices
+
+    fill_results = []
+    for idx, i in enumerate(mask_token_index):
+        for j, token_idx in enumerate(topk_tokens[idx]):
+            token_text = tokenizer.convert_ids_to_tokens(token_idx.item())
+            token_text = token_text.replace("Ġ", " ")  # Replace special characters with a space
+            probability = torch.softmax(predictions.logits[0, i], dim=-1)[token_idx].item()
+            fill_results.append({
+                'score': probability,
+                'token': token_idx.item(),
+                'token_str': token_text,
+                'sequence': tokenized_text.replace("<mask>", token_text.strip())
+            })
+
+    fill_results.sort(key=lambda x: x['score'], reverse=True)
+    return fill_results
+
+# Example usage of the function
+text = "Včera večer sme <mask> nový film v kine, ktorý mal premiéru iba pred týždňom."
+result = fill_mask(text.lower(), tokenizer, model, top_k=5)
+print(result)
+
+[{'score': 0.4014046788215637,
+  'token': 6626,
+  'token_str': ' videli',
+  'sequence': 'včera večer sme videli nový film v kine, ktorý mal premiéru iba pred týždňom.'},
+ {'score': 0.15018892288208008,
+  'token': 874,
+  'token_str': ' mali',
+  'sequence': 'včera večer sme mali nový film v kine, ktorý mal premiéru iba pred týždňom.'},
+ {'score': 0.057530131191015244,
+  'token': 21193,
+  'token_str': ' pozreli',
+  'sequence': 'včera večer sme pozreli nový film v kine, ktorý mal premiéru iba pred týždňom.'},
+ {'score': 0.049020398408174515,
+  'token': 26468,
+  'token_str': ' sledovali',
+  'sequence': 'včera večer sme sledovali nový film v kine, ktorý mal premiéru iba pred týždňom.'},
+ {'score': 0.04107135161757469,
+  'token': 9171,
+  'token_str': ' objavili',
+  'sequence': 'včera večer sme objavili nový film v kine, ktorý mal premiéru iba pred týždňom.'}]
+```
+
+## Training Data
+
+The `SK_Morph_BLM` model was pretrained using a subset of the OSCAR 2019 corpus, specifically focusing on the Slovak language. The corpus underwent comprehensive preprocessing to ensure the quality and relevance of the data:
+
+- **Language Filtering:** Non-Slovak text was removed to focus solely on the Slovak language.
+- **Character Normalization:** Various types of spaces, quotes, dashes, and separators were standardized (e.g., replacing different types of spaces with a single space, or dashes with hyphens). Emoticons were replaced with spaces.
+- **Symbol and Unwanted Text Removal:** Sentences containing mathematical symbols, pictograms, or characters from Asian and African languages were deleted. Duplicates of punctuation, special characters, and spaces were also removed.
+- **URL and Text Normalization:** All web addresses were removed, and the text was converted to lowercase to simplify tokenization.
+- **Content Cleanup:** Text that included irrelevant content from web crawling, such as keywords and HTML tags, was identified and removed.
+
+Additionally, the preprocessing included further refinement steps to create the final dataset:
+
+- **Parentheses Content Removal:** All content within parentheses was removed to reduce noise.
+- **Selection of Text Segments:** Medium-length text paragraphs were selected to maintain consistency.
+- **Similarity Filtering:** Paragraphs with at least 50% similarity to previous ones were removed to minimize redundancy.
+- **Random Sampling:** Finally, 20% of the remaining paragraphs were randomly selected.
+
+After preprocessing, the training corpus consisted of:
+- **455 MB of text**
+- **895,125 paragraphs**
+- **64.6 million words**
+- **1.13 million unique words**
+- **119 unique characters**
+
+## Pretraining
+
+The `SK_Morph_BLM` model was trained with the following key parameters:
+
+- **Architecture:** Based on RoBERTa, with 6 hidden layers and 12 attention heads.
+- **Hidden size:** 576
+- **Vocabulary size:** 50,264 tokens
+- **Sequence length:** 256 tokens
+- **Dropout:** 0.1
+- **Number of parameters:** 58 million
+- **Optimizer:** AdamW, learning rate 1×10^(-4), weight decay 0.01
+- **Training:** 30 epochs, divided into 3 phases:
+  - **Phase 1:** 10 epochs on CPU (4x AMD EPYC 7542), batch size 64, 50 hours per epoch, 139,870 steps total.
+  - **Phase 2:** 5 epochs on GPU (1x Nvidia A100 40GB), batch size 64, 100 minutes per epoch, 69,935 steps total.
+  - **Phase 3:** 15 epochs on GPU (2x Nvidia A100 40GB), batch size 128, 60 minutes per epoch, 104,910 steps total.
+
+The model was trained using the Hugging Face library, but without using the `Trainer` class—native PyTorch was used instead.
+
+## Fine-Tuned Versions of the SK_Morph_BLM Model
+
+Here are the fine-tuned versions of the `SK_Morph_BLM` model based on the folders provided:
+
+- [`SK_Morph_BLM-ner`](https://huggingface.co/daviddrzik/SK_Morph_BLM-ner): Fine-tuned for Named Entity Recognition (NER) tasks.
+- [`SK_Morph_BLM-pos`](https://huggingface.co/daviddrzik/SK_Morph_BLM-pos): Fine-tuned for Part-of-Speech (POS) tagging.
+- [`SK_Morph_BLM-qa`](https://huggingface.co/daviddrzik/SK_Morph_BLM-qa): Fine-tuned for Question Answering tasks.
+- [`SK_Morph_BLM-sentiment-csfd`](https://huggingface.co/daviddrzik/SK_Morph_BLM-sentiment-csfd): Fine-tuned for sentiment analysis on the CSFD (movie review) dataset.
+- [`SK_Morph_BLM-sentiment-multidomain`](https://huggingface.co/daviddrzik/SK_Morph_BLM-sentiment-multidomain): Fine-tuned for sentiment analysis across multiple domains.
+- [`SK_Morph_BLM-sentiment-reviews`](https://huggingface.co/daviddrzik/SK_Morph_BLM-sentiment-reviews): Fine-tuned for sentiment analysis on general review datasets.
+- [`SK_Morph_BLM-topic-news`](https://huggingface.co/daviddrzik/SK_Morph_BLM-topic-news): Fine-tuned for topic classification in news articles.