README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - question-answering
datasets:
  - squad
  - newsqa
  - hotpotqa
  - searchqa
  - triviaqa-web
  - naturalquestions
  - qamr
  - duorc
  - google/boolq
  - commonsense_qa
  - hellaswag
  - race
  - social_i_qa
  - drop
  - narrativeqa
  - hybrid_qa
metrics:
  - squad
  - accuracy

Description

Checkpoint of MetaQA from MetaQA: Combining Expert Agents for Multi-Skill Question Answering (https://arxiv.org/abs/2112.01922)

How to Use

from inference import MetaQA, PredictionRequest
metaqa = MetaQA("haritzpuerto/MetaQA")
# run the QA Agents with the input question and context. For this example, I will show mockup outputs from extractive QA agents.
list_preds = [('Utah', 0.1442876160144806),
                ('DOC] [TLE] 1886', 0.10822545737028122),
                ('Utah Territory', 0.6455602645874023),
                ('Eli Murray opposed the', 0.352359801530838),
                ('Utah', 0.48052430152893066),
                ('Utah Territory', 0.35186105966567993),
                ('Utah', 0.8328599333763123),
                ('Utah', 0.3405868709087372),
            ]
# add ("", 0.0) to the list of predictions until the size is 16 (because MetaQA was trained on 16 datasets/agents including other formats, not only extractive)
for i in range(16-len(list_preds)):
    list_preds.append(("", 0.0))

request = PredictionRequest()
request.input_question = "While serving as Governor of this territory, 1880-1886, Eli Murray opposed the advancement of polygamy?"
request.input_predictions = list_preds

(pred, agent_name, metaqa_score, agent_score) = metaqa.run_metaqa(request)