--- license: apache-2.0 tags: - generated_from_trainer - text generation - email generation - email datasets: - aeslc - postbot/multi-emails-100k widget: - text: 'Good Morning Professor Beans, Hope you are doing well. I just wanted to reach out and ask if differential calculus will be on the exam' example_title: email to prof - text: 'Hey , Thank you for signing up for my weekly newsletter. Before we get started, you''ll have to confirm your email address.' example_title: newsletter - text: 'Hi , I hope this email finds you well. I wanted to reach out and ask about office hours' example_title: office hours - text: 'Greetings , I hope you had a splendid evening at the Company sausage eating festival. I am reaching out because' example_title: festival - text: 'Good Morning Harold, I was wondering when the next' example_title: event - text: URGENT - I need the TPS reports example_title: URGENT - text: 'Hi Archibald, I hope this email finds you extremely well.' example_title: emails that find you - text: 'Hello there. I just wanted to reach out and check in to' example_title: checking in - text: 'Hello , I hope this email finds you well. I wanted to reach out and see if you''ve enjoyed your time with us' example_title: work well - text: 'Hi , I hope this email finds you well. I wanted to reach out and see if we could catch up' example_title: catch up - text: I'm and I just moved into the area and wanted to reach out and get some details on where I could get groceries and example_title: grocery parameters: min_length: 32 max_length: 128 no_repeat_ngram_size: 2 do_sample: true temperature: 0.4 top_k: 30 top_p: 0.9 repetition_penalty: 3.5 length_penalty: 0.9 base_model: EleutherAI/gpt-neo-1.3B model-index: - name: gpt-neo-1.3B-emailgen results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 29.95 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 47.95 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 24.11 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 42.55 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 56.27 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 0.0 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=postbot/gpt-neo-1.3B-emailgen name: Open LLM Leaderboard --- # gpt-neo-1.3B-emailgen This model is a fine-tuned version of [EleutherAI/gpt-neo-1.3B](https://huggingface.co/EleutherAI/gpt-neo-1.3B) on the postbot/multi-emails-100k dataset. It achieves the following results on the evaluation set: - Loss: 1.6930 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - distributed_type: multi-GPU - gradient_accumulation_steps: 32 - total_train_batch_size: 128 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.02 - num_epochs: 2 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 1.8669 | 1.0 | 789 | 1.7866 | | 1.4049 | 2.0 | 1578 | 1.6930 | ### Framework versions - Transformers 4.22.2 - Pytorch 1.10.0+cu113 - Tokenizers 0.12.1 # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_postbot__gpt-neo-1.3B-emailgen) | Metric |Value| |---------------------------------|----:| |Avg. |33.47| |AI2 Reasoning Challenge (25-Shot)|29.95| |HellaSwag (10-Shot) |47.95| |MMLU (5-Shot) |24.11| |TruthfulQA (0-shot) |42.55| |Winogrande (5-shot) |56.27| |GSM8k (5-shot) | 0.00|