| # GPT2-Horoscopes | |
| [](https://share.streamlit.io/shahp7575/gpt2-horoscopes-app/generate.py) | |
| ## Model Description | |
| GPT2 fine-tuned on Horoscopes dataset scraped from [Horoscopes.com](https://www.horoscope.com/us/index.aspx). This model generates horoscopes given a horoscope *category*. | |
| ## Uses & Limitations | |
| ### How to use | |
| The model can be used directly with the HuggingFace `pipeline` API. | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelWithLMHead | |
| tokenizer = AutoTokenizer.from_pretrained("shahp7575/gpt2-horoscopes") | |
| model = AutoModelWithLMHead.from_pretrained("shahp7575/gpt2-horoscopes") | |
| ``` | |
| ### Generation | |
| Input Text Format - `<|category|> {category_type} <|horoscope|>` | |
| Supported Categories - *general, career, love, wellness, birthday* | |
| Example: | |
| ```python | |
| prompt = <|category|> career <|horoscope|> | |
| prompt_encoded = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0) | |
| sample_outputs = model.generate(prompt, | |
| do_sample=True, | |
| top_k=40, | |
| max_length = 300, | |
| top_p=0.95, | |
| temperature=0.95, | |
| num_return_sequences=1) | |
| ``` | |
| For reference this [generation script](https://github.com/shahp7575/gpt2-horoscopes/blob/master/generate_from_hub.py) can be used as well. | |
| ### Training Data | |
| Dataset is scraped from [Horoscopes.com](https://www.horoscope.com/us/index.aspx) for 5 categories with a total of ~12k horoscopes. The dataset can be found on [Kaggle](https://www.kaggle.com/shahp7575/horoscopes). | |
| ### Training Procedure | |
| The model uses the [GPT2](https://huggingface.co/gpt2) checkpoint and then is fine-tuned on horoscopes dataset for 5 different categories. Since the goal of the fine-tuned model was also to understand different horoscopes for different category types, the *categories* are added to the training data separated by special token `<|category|>`. | |
| **Training Parameters:** | |
| - EPOCHS = 5 | |
| - LEARNING RATE = 5e-4 | |
| - WARMUP STEPS = 1e2 | |
| - EPSILON = 1e-8 | |
| - SEQUENCE LENGTH = 300 | |
| ### Evaluation Results | |
| Loss: 2.77 | |
| ### Limitations | |
| This model is only fine-tuned on horoscopes by categories. They do not, and neither attempt to, represent actual horoscopes. It is developed only for educational and learning purposes. | |
| ## References | |
| - [Rey Farhan's - Fine-tuning GPT2 Notebook](https://colab.research.google.com/drive/13dZVYEOMhXhkXWfvSMVM1TTtUDrT6Aeh?usp=sharing#scrollTo=_U3m6wr3Ahzt) | |
| - [Jonathan Bgn - Building a Slogan Generator with GPT-2](https://jonathanbgn.com/gpt2/2020/01/20/slogan-generator.html) |