maritaca-ai

/

sabia-7b

like 76

Hi @vabatista , this model was trained only on a next-token prediction task, it wasn't finetuned to follow instructions. Therefore, it will only work if you include few-shot examples in the prompt.

Thanks for quick answer!

Could you provide a snippet of text you used to assess, for instance, the faquad benchmark?

This one was used for Faquad:

Answer the question based on the context.

Example 1:
Context: Complementary rules for registration renewal. In the online course selection stage, the student must fill out an electronic form, through the Academic System (Siscad),
indicating the subjects he intends to take in the subsequent academic period, respecting the prerequisites and the compatibility of class schedules of the selected subjects.
Place priority is not guaranteed to students who have attended any prerequisites in the immediately preceding summer or winter term or who, for any reason, do not select
the respective subject(s) at this stage of the registration renewal.
Question: Which subjects must the academic fill in the electronic form?
Answer: subjects he intends to take in the subsequent academic period
###
Example 2:
Context: ...
Question: ...
Answer: ...
###
Example 3:
Context: ...
Question: ...
Answer: 

In case you want to check for more examples, see appendix A of the paper https://arxiv.org/pdf/2304.07880v3.pdf

Closing this for now but please let me know if you need any other information.