Spaces:

RMT-team

/

babilong

like 11

Running

The following results added:

• 0k on qa1-20 + avg
• 4k on qa1-10 + avg
• 8k on qa1-10 + avg

Models used:

• Llama 3 8b Instruct FP16
• Llama 3 70b Instruct FP16
• Qwen 2.5 72b Instruct FP16

To compute Exact Match scores, both target and response were first lowered.

The following prompts were used:

SYSTEM_PROMPT = "You are a intelligent assistant."
SPECIFIC_PROMPTS = {
    'qa1': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa2': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa3': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa4': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa5': 'Answer with a single word contatining name of the person or the object. <answer example>Jeff</answer example> <answer example>football</answer example>',
    'qa6': 'Answer with a single word `yes` or `no`. <answer example>yes</answer example>',
    'qa7': 'Answer with a single word containing number of objects or `none`. <answer example>one</answer example> <answer example>none</answer example>',
    'qa8': 'Answer with words containing objects or `nothing`. <answer example>nothing</answer example> <answer example>apple</answer example> <answer example>apple,football</answer example>',
    'qa9': 'Answer with a single word `yes` or `no`. <answer example>yes</answer example>',
    'qa10': 'Answer with a single word `yes` or `no` or `maybe`. <answer example>maybe</answer example>',
    'qa11': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa12': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa13': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa14': 'Answer with a single word contatining location of the person. <answer example>bedroom</answer example>',
    'qa15': 'Answer with a single word contatining animal. <answer example>wolf</answer example>',
    'qa16': 'Answer with a single word contatining color. <answer example>white</answer example>',
    'qa17': 'Answer with a single word `yes` or `no`. <answer example>yes</answer example>',
    'qa18': 'Answer with a single word `yes` or `no`. <answer example>yes</answer example>',
    'qa19': 'Answer with a direction pairs containing a combination of two movements. <answer example>s,w</answer example> <answer example>e,e</answer example> <answer example>n,e</answer example>',
    'qa20': 'Answer with a single word containing location of the person or the condition of the person. <answer example>bedroom</answer example> <answer example>tired</answer example>',
}
def generate_search_prompt(item, task):
    return f"""
You will be provided with context that includes relevant facts and a question.
You need to answer the question based only on the information from the facts.

<context>
{item["input"]}
</context>

QUESTION: {item["question"]}

Always return your direct answer to the question. {SPECIFIC_PROMPTS[task]}
Do not write anything else after that.
"""