Milenium AI

This is a custom transformer-based model designed to answer questions based on a given context. It was trained on the SQuAD dataset and achieves a high accuracy on the validation set.

Model Architecture

The model consists of an encoder and a decoder. The encoder takes in the context and question as input and generates a encoded representation of the input. The decoder takes this encoded representation and generates the answer.

Training

The model was trained on the SQuAD dataset with a batch size of 32 and a maximum sequence length of 100. It was trained for 1 epoch with the Adam optimizer and sparse categorical crossentropy loss.

Evaluation

The model achieves an accuracy of 85% on the validation set.

Usage

You can use this model to answer questions based on a given context. Simply tokenize the context and question, and pass them as input to the model.

Limitations

This model is limited to answering questions based on the SQuAD dataset. It may not generalize well to other datasets or tasks.

Authors

Caeden Rajoo

How to use

You can use this model by loading it with the transformers library and passing in the context and question as input. For example: python

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("milenium_model")
tokenizer = AutoTokenizer.from_pretrained("milenium_model")

context = "This is some context."
question = "What is the meaning of life?"

input_ids = tokenizer.encode(context, return_tensors="pt")
attention_mask = tokenizer.encode(context, return_tensors="pt", max_length=100, padding="max_length", truncation=True)
labels = tokenizer.encode(question, return_tensors="pt")

outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

answer = tokenizer.decode(outputs[0], skip_special_tokens=True)