Cheapity3 🐷

GPT3-like T5 model trained to generate text in multiple languages.

Motivation

GPT models are expensive run.
GPT models are monolingual.

Solution

Maybe, Small Models aren't Terrible (SMarT)
Plus, they are cheaper to run.

I fine-tuned T5 on multiple languages (🇬🇧 English, 🇩🇪 German, 🇫🇷 French) and multiple academic text snippets from various domains like tech, law, finance and science etc. to generate text, just like GPT models do.

Usage

Provide some text e.g "Italy, officially the Italian Republic is a country consisting of"
Tell Cheapity3 how many words you want to generate e.g 15 -- 😃 Yes, you can control the length.
Cheapity3 reads your text and generates a continuation containing approximately 15 words.

from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("flexudy/cheapity3")

model = AutoModelWithLMHead.from_pretrained("flexudy/cheapity3")

input_text = """The mechanical engineering field requires an understanding of core areas including mechanics, dynamics,
 thermodynamics, materials science, structural analysis, and 
 electricity. { _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ }"""  # 15 words

inputs = tokenizer(input_text, return_tensors="pt", truncation=True, max_length=512)

input_ids = inputs["input_ids"]

attention_mask = inputs["attention_mask"]

outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    max_length=128,
    do_sample=True,
    early_stopping=True,
    num_return_sequences=4,
    repetition_penalty=2.5
)

for i in range(4):
    print(tokenizer.decode(outputs[i], skip_special_tokens=True, clean_up_tokenization_spaces=True))

# Italy, officially the Italian Republic is a country consisting of 
# > Cheapity3 continuing:
# ... Developing the knowledge base for these core areas will enable engineers to build their capabilities rapidly and efficiently. ...
# ... The field of mechanics offers a variety and broad range for applications throughout the engineering/technological fields. ...
# ... Mechanics generally is not understood by students. While they can be employed in the field, mechanical engineering ...
# ... Introduction to mechanical engineering and core fields including chemical products, materials science, structural analysis, and geomatics ...