metadata

license: apache-2.0
datasets:
  - Dahoas/synthetic-instruct-gptj-pairwise
  - databricks/databricks-dolly-15k
  - HuggingFaceH4/instruction-dataset
  - nicholasKluge/instruct-aira-dataset
language:
  - en
metrics:
  - bleu
library_name: transformers
tags:
  - alignment
  - instruction tuned
  - text generation
  - conversation
  - assistant
pipeline_tag: text-generation
widget:
  - text: <|startoftext|>Hello! What is your name?<|endoftext|>
    example_title: Greetings
  - text: <|startoftext|>Can you explain what is Machine Learning?<|endoftext|>
    example_title: Machine Learning
  - text: <|startoftext|>Do you know anything about virtue ethics?<|endoftext|>
    example_title: Ethics
  - text: <|startoftext|>How can I make my girlfried happy?<|endoftext|>
    example_title: Advise
inference:
  parameters:
    repetition_penalty: 1.2
    temperature: 0.2
    top_k: 30
    top_p: 0.3
    max_length: 200
    length_penalty: 0.3
    early_stopping: true
model-index:
  - name: Aira-Instruct-774M
    results:
      - task:
          type: text-generation
        dataset:
          type: text-generation
          name: truthful_qa
        metrics:
          - name: rouge
            type: rouge
            value: 0.23884372491125055
            verified: false
co2_eq_emissions:
  emissions: 3
  source: CodeCarbon
  training_type: fine-tuning
  geographical_location: Canada
  hardware_used: NVIDIA A100-SXM4-40GB

Aira-Instruct-774M

Aira-Instruct-774M is a instruction-tuned GPT-style model based on GPT-2. The model was trained with a dataset composed of prompt, completions, generated via the Self-Instruct framework. Aira-Instruct-774M instruction-tuning was achieved via conditional text generation.

The dataset used to train this model combines the following sources of data: the synthetic-instruct-gptj-pairwise dataset, the databricks_dolly_15k dataset, the instruction-dataset dataset, and a subset of Aira's fine-tuning dataset, focused on Q&A related to Ethics, AI, AI safety, and other related topics. The dataset is available in both Portuguese and English.

Check our gradio-demo in Spaces.

Details

Size: 774,032,640 parameters
Dataset: Instruct-Aira Dataset
Language: English
Number of Epochs: 3
Batch size: 8
Optimizer: torch.optim.AdamW (warmup_steps = 1e2, learning_rate = 5e-4, epsilon = 1e-8)
GPU: 1 NVIDIA A100-SXM4-40GB
Emissions: 0.0030 KgCO2 (Canada)
Total Energy Consumption: 1.29 kWh

Epoch/Loss	Training	Validation
1	0.696885	0.638819
2	0.516360	0.610071
3	0.338896	0.647381

This repository has the notebook used to train this model.

Usage

Two special tokens are used to mark the user side of the interaction and the model's response:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained('nicholasKluge/Aira-Instruct-774M')
aira = AutoModelForCausalLM.from_pretrained('nicholasKluge/Aira-Instruct-774M')

aira.eval()
aira.to(device)

question =  input("Enter your question: ")

inputs = tokenizer(tokenizer.bos_token + question + tokenizer.eos_token, return_tensors="pt").to(device)

responses = aira.generate(**inputs,
    bos_token_id=tokenizer.bos_token_id,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
    do_sample=True,
    top_k=50,
    max_length=200,
    top_p=0.95,
    temperature=0.7,
    num_return_sequences=2)

print(f"Question: 👤 {question}\n")

for i, response in  enumerate(responses):
    print(f'Response {i+1}: 🤖 {tokenizer.decode(response, skip_special_tokens=True).replace(question, "")}')

The model will output something like:

>>> Question: 👤 Hello! What is your name?

>>>Response 1: 🤖 Hi there! I am Aira, a chatbot designed to answer questions about AI ethics and AI safety. If you need assistance navigating our conversation, please feel free to ask!
>>>Response 2: 🤖 Hi there! My name is Aira, and I'm a chatbot designed to answer questions related to AI ethics and AI Safety. If you need assistance, feel free to ask, and I'll be happy to help you out.

Limitations

🤥 Generative models can perpetuate the generation of pseudo-informative content, that is, false information that may appear truthful.

🤬 In certain types of tasks, generative models can produce harmful and discriminatory content inspired by historical stereotypes.

Cite as 🤗


@misc{nicholas22aira,
  doi = {10.5281/zenodo.6989727},
  url = {https://huggingface.co/nicholasKluge/Aira-Instruct-774M},
  author = {Nicholas Kluge Corrêa and Carolina Del Pino},
  title = {Aira},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
}

License

The Aira-Instruct-774M is licensed under the Apache License, Version 2.0. See the LICENSE file for more details.