Repository: EleutherAI/lm-evaluation-harness
Issue: DROP benchmark prompt issue
Root cause: Configuration Error

Hi, I think there is an issue with the DROP in current implementation. 
The scores for the DROP task are nearly zero, and after checking the prompt, it seems that there is an issue with it.

```bash
python -m scripts.write_out \
    --output_base_path ./output_samples \
    --tasks drop \
    --num_fewshot 3 \
    --sets val \
    --num_examples 2
```

<details>
<summary>3-shot output in current implementation</summary>

```
!!@@##@@!! -- Example 0
Hoping to rebound from their loss to the Titans, the Cowboys flew to the Hubert H. Humphrey Metrodome for a Week 6 duel with the Minnesota Vikings.  Dallas delivered the opening punch in the first quarter as quarterback Tony Romo found wide receiver Roy Williams on a 15-yard touchdown pass.  The Vikings answered with quarterback Brett Favre completed a 10-yard touchdown pass to wide receiver Greg Camarillo.  In the second quarter, the Cowboys regained the lead as Romo and Williams hooked up with each other again on a 2-yard touchdown pass. Minnesota took the lead in the third quarter as wide receiver Percy Harvin returned the half's opening kickoff 95 yards for a touchdown, followed by running back Adrian Peterson's 1-yard touchdown run.  In the fourth quarter, Dallas tied the game as Romo connected with rookie wide receiver Dez Bryant on a 31-yard touchdown pass.  The Vikings responded with kicker Ryan Longwell making a 38-yard field goal. With only 5 seconds left in the game, the Cowboys attempted a lateral play, in a fashion similar to The Play, River City Relay, and the Music City Miracle. However, Romo was then flagged for throwing a forward pass after crossing the line of scrimmage, ending the game. Who scored first in the game against Minnesota?number,date,spans,worker_id,hit_id

The 49ers dominated the first half and led 28-7 at halftime.  However, their second half offense stalled while the defense fell apart as the 49ers squandered a 21-point 3rd quarter lead and a 14-point lead in the final minutes of the 4th quarter.  The Chargers outscored the 49ers 31-7 after halftime.  Trailing 35-28, the Chargers drove down the field for the game-tying touchdown, converting two 4th downs on that drive.  The 49ers attempted to win in the final seconds of regulation, but Phil Dawson's 60 yard kick was way off.  The game went to overtime.  The 49ers got the ball first, but a fumble by Quinton Patton allowed the Chargers to have possession on their own 40.  The Chargers drove the 49ers' 22 yard line before kicking a field goal to hand the 49ers their fourth straight loss. The 49ers rushed for an astonishing 355 yards, including a 52-yard touchdown run by Frank Gore and a 90-yard touchdown run by Colin Kaepernick. The 49ers set an NFL record for most rushing yards in a loss The 49ers also scored 35 points, a season high.  However, their offensive struggles in the second half of games continued.  During the four game losing streak, the 49ers scored only 13 points in the second half of their games (outscored 65-13 by opponents in the second half of those games).  With the loss the 49ers fell to 7-8, ensuring that they would finish without a winning record for the first time since the 2010 season. Which team fell behind in the first half?number,date,spans,worker_id,hit_id

Coming off their bye week, the Cardinals went home for a Week 7 interconference duel with the Pittsburgh Steelers, in a rematch of Super Bowl XLIII.  Arizona trailed early in the first quarter as Steelers quarterback Ben Roethlisberger completed a 12-yard touchdown pass to tight end Heath Miller.  Pittsburgh would add onto their lead in the second quarter as  Roethlisberger completed a 95-yard touchdown pass to wide receiver Mike Wallace.  The Cardinals would answer with a 1-yard touchdown run from running back Alfonso Smith.  The Steelers would close out the half with kicker Shaun Suisham getting a 41-yard field goal. Arizona opened the third quarter with quarterback Kevin Kolb finding running back LaRod Stephens-Howling on a 73-yard touchdown pass, but Pittsburgh struck back with Roethlisberger completing a 4-yard touchdown pass to wide receiver Emmanuel Sanders, along with Kolb's intentional grounding penalty in the endzone resulting in a safety.  In the fourth quarter, the Steelers continued to pull away with Suisham making a 42-yard and a 39-yard field goal.  The Cardinals tried to rally as Kolb found wide receiver Early Doucet on a 2-yard touchdown pass (with a failed two-point conversion), but Pittsburgh held on to preserve the victory. How many touchdowns did Roethlisberger throw?number,date,spans,worker_id,hit_id

 Hoping to rebound from their loss to the Patriots, the Raiders stayed at home for a Week 16 duel with the Houston Texans.  Oakland would get the early lead in the first quarter as quarterback JaMarcus Russell completed a 20-yard touchdown pass to rookie wide receiver Chaz Schilens.  The Texans would respond with fullback Vonta Leach getting a 1-yard touchdown run, yet the Raiders would answer with kicker Sebastian Janikowski getting a 33-yard and a 30-yard field goal.  Houston would tie the game in the second quarter with kicker Kris Brown getting a 53-yard and a 24-yard field goal. Oakland would take the lead in the third quarter with wide receiver Johnnie Lee Higgins catching a 29-yard touchdown pass from Russell, followed up by an 80-yard punt return for a touchdown.  The Texans tried to rally in the fourth quarter as Brown nailed a 40-yard field goal, yet the Raiders' defense would shut down any possible attempt. Who scored the first touchdown of the game?
```

</details>


It seems [doc_to_target](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/drop/default.yaml#L8) converts all the targets as `number,date,spans,worker_id,hit_id` in current implementation.

```
# after process docs
>>> ds_['validation'][0]
{'section_id': 'nfl_1184', 
'passage': " Hoping to rebound from their loss to the Patriots, the Raiders stayed at home for a Week 16 duel with the Houston Texans.  Oakland would get the early lead in the first quarter as quarterback JaMarcus Russell completed a 20-yard touchdown pass to rookie wide receiver Chaz Schilens.  The Texans would respond with fullback Vonta Leach getting a 1-yard touchdown run, yet the Raiders would answer with kicker Sebastian Janikowski getting a 33-yard and a 30-yard field goal.  Houston would tie the game in the second quarter with kicker Kris Brown getting a 53-yard and a 24-yard field goal. Oakland would take the lead in the third quarter with wide receiver Johnnie Lee Higgins catching a 29-yard touchdown pass from Russell, followed up by an 80-yard punt return for a touchdown.  The Texans tried to rally in the fourth quarter as Brown nailed a 40-yard field goal, yet the Raiders' defense would shut down any possible attempt.", 'question': 'Who scored the first touchdown of the game?', 'query_id': 'f37e81fa-ef7b-4583-b671-762fc433faa9', 
'answer': {'number': '', 'date': {'day': '', 'month': '', 'year': ''}, 'spans': ['Chaz Schilens'], 'worker_id': '', 'hit_id': ''}, 
'validated_answers': {'number': ['', ''], 'date': [{'day': '', 'month': '', 'year': ''}, {'day': '', 'month': '', 'year': ''}], 
'spans': [['Chaz Schilens'], ['JaMarcus Russell']], 
'worker_id': ['', ''], 'hit_id': ['', '']}, 'id': 'f37e81fa-ef7b-4583-b671-762fc433faa9', 
'answers': [['Chaz Schilens'], ['JaMarcus Russell']]}
```

I'm not familiar with the DROP task, so I'm not sure about the details. 
However, based on the code [before the big refactor](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/drop.py#L123-L124), it seems that doc_to_target should be something like `" {{ answers[0]|join(', ')}}"`. 
If you could check this, I could submit a PR.


No way this few-shot prompt works...

![Image](https://github.com/user-attachments/assets/265eca54-aa94-4247-91f7-5613aaf4835b)

cc @baberabb 

After digging deep into the implementation, I managed to get it work and match the performance reported by `meta-llama/Meta-Llama-3-8B`

![Image](https://github.com/user-attachments/assets/a81c4b84-b2cf-485f-bec8-bece8bfd71da)

![Image](https://github.com/user-attachments/assets/522c9a08-e65b-4bd8-b8b2-bd25f6f55d88)

---
for `utils.py`
```python
import re
import string

import numpy as np


_ARTICLES = re.compile(r"\b(a|an|the)\b", re.UNICODE)


def process_docs(dataset):
    def _process(doc):
        return {
            "id": doc["query_id"],
            "passage": doc["passage"],
            "question": doc["question"],
            "answers": get_answers(doc),
        }

    return dataset.map(_process)

def get_answers(doc):
    answers = []
    
    main_answer = parse_answer(doc["answer"])
    answers.extend(main_answer)
    
    validated = doc["validated_answers"]
    for i in range(len(validated["number"])):
        candidate = {
            "number": validated["number"][i],
            "date": validated["date"][i],
            "spans": validated["spans"][i]
        }
        validated_answer = parse_answer(candidate)
        answers.extend(validated_answer)
    
    seen = set()
    unique_answers = []
    for ans in answers:
        if ans not in seen:
            seen.add(ans)
            unique_answers.append(ans)
    
    return unique_answers

# def get_answers(doc):
#     def _flatten_validated_answers(validated_answers):
#         """Flattens a dict of lists of validated answers.
#         {"number": ['1', '8'], ...}
#         -> [{"number": ['1'], ...}, {"number": ['8'], ...}]
#         """
#         valid_answers = []
#         for i in range(len(validated_answers["number"])):
#             valid_answers.append(
#                 {
#                     "number": validated_answers["number"][i],
#                     "date": validated_answers["date"][i],
#                     "spans": validated_answers["spans"][i],
#                 }
#             )
#         return valid_answers

#     answers = []
#     answers_set = set()
#     candidates = [doc["answer"]] + _flatten_validated_answers(doc["validated_answers"])
#     for candidate in candidates:
#         answer = parse_answer(candidate)
#         if answer in answers_set:
#             continue
#         answers_set.add(answer)
#         answers.append(answer[0])
#     return answers


def parse_answer(answer):
    # NOTE: Everything is returned as a tuple for uniformity and hashability.
    if answer["number"] != "":
        return (str(answer["number"]),)
    if answer["spans"] != []:
        return tuple(answer["spans"])
    return (
        " ".join(
            [answer["date"]["day"], answer["date"]["month"], answer["date"]["year"]]
        ).strip(),
    )


def process_results(doc, results):
    preds, golds = results, doc["answers"]
    
    max_em = 0
    max_f1 = 0
    for gold_answer in golds:
        exact_match, f1_score = get_metrics(preds, gold_answer)
        if gold_answer[0].strip():
            max_em = max(max_em, exact_match)
            max_f1 = max(max_f1, f1_score)
    return {"em": max_em, "f1": max_f1}


def get_metrics(predicted, gold):
    """
    Takes a predicted answer and a gold answer (that are both either a string or a list of
    strings), and returns exact match and the DROP F1 metric for the prediction.  If you are
    writing a script for evaluating objects in memory (say, the output of predictions during
    validation, or while training), this is the function you want to call, after using
    :func:`answer_json_to_strings` when reading the gold answer from the released data file.
    """
    predicted_bags = _answer_to_bags(predicted)
    gold_bags = _answer_to_bags(gold)

    if set(predicted_bags[0]) == set(gold_bags[0]) and len(predicted_bags[0]) == len(
        gold_bags[0]
    ):
        exact_match = 1.0
    else:
        exact_match = 0.0

    f1_per_bag = _align_bags(predicted_bags[1], gold_bags[1])
    f1 = np.mean(f1_per_bag)
    f1 = round(f1, 2)
    return exact_match, f1


def _answer_to_bags(answer):
    if isinstance(answer, (list, tuple)):
        raw_spans = answer
    else:
        raw_spans = [answer]
    normalized_spans = []
    token_bags = []
    for raw_span in raw_spans:
        normalized_span = _normalize(raw_span)
        normalized_spans.append(normalized_span)
        token_bags.append(set(normalized_span.split()))
    return normalized_spans, token_bags


def _align_bags(predicted, gold):
    """
    Takes gold and predicted answer sets and first finds the optimal 1-1 alignment
    between them and gets maximum metric values over all the answers.
    """
    from scipy.optimize import linear_sum_assignment

    scores = np.zeros([len(gold), len(predicted)])
    for gold_index, gold_item in enumerate(gold):
        for pred_index, pred_item in enumerate(predicted):
            if _match_numbers_if_present(gold_item, pred_item):
                scores[gold_index, pred_index] = _compute_f1(pred_item, gold_item)
    row_ind, col_ind = linear_sum_assignment(-scores)

    max_scores = np.zeros([max(len(gold), len(predicted))])
    for row, column in zip(row_ind, col_ind):
        max_scores[row] = max(max_scores[row], scores[row, column])
    return max_scores


def _compute_f1(predicted_bag, gold_bag):
    intersection = len(gold_bag.intersection(predicted_bag))
    if not predicted_bag:
        precision = 1.0
    else:
        precision = intersection / float(len(predicted_bag))
    if not gold_bag:
        recall = 1.0
    else:
        recall = intersection / float(len(gold_bag))
    f1 = (
        (2 * precision * recall) / (precision + recall)
        if not (precision == 0.0 and recall == 0.0)
        else 0.0
    )
    return f1


def _match_numbers_if_present(gold_bag, predicted_bag):
    gold_numbers = set()
    predicted_numbers = set()
    for word in gold_bag:
        if _is_number(word):
            gold_numbers.add(word)
    for word in predicted_bag:
        if _is_number(word):
            predicted_numbers.add(word)
    if (not gold_numbers) or gold_numbers.intersection(predicted_numbers):
        return True
    return False


def _is_number(text):
    try:
        float(text)
        return True
    except ValueError:
        return False


def _remove_articles(text):
    return _ARTICLES.sub(" ", text)


def _white_space_fix(text):
    return " ".join(text.split())


def _remove_punc(text):
    exclude = set(string.punctuation) - {"."}
    if not _is_number(text):
        return "".join(ch for ch in text if ch not in exclude)
    else:
        return text


def _fix_number(text):
    return str(float(text)) if _is_number(text) else text


def _tokenize(text):
    return re.split(" |-", text)


def _normalize(answer):
    tokens = [
        _white_space_fix(_remove_articles(_fix_number(_remove_punc(token.lower()))))
        for token in _tokenize(answer)
    ]
    tokens = [token for token in tokens if token.strip()]
    normalized = " ".join(tokens).strip()
    return normalized

```

for `config.yaml`
```yaml
task: my_drop
dataset_path: EleutherAI/drop
output_type: generate_until
training_split: train
validation_split: validation
process_docs: !function utils.process_docs
doc_to_text: "Passage: {{passage}}\nQuestion: {{question}}\nAnswer:"
doc_to_target: "{{ answers|join(', ')}}"
target_delimiter: " "
process_results: !function utils.process_results
should_decontaminate: true
doc_to_decontamination_query: "{{passage}} {{question}}"
generation_kwargs:
  until:
    - "\n"
metric_list:
  - metric: em
    aggregation: mean
    higher_is_better: true
  - metric: f1
    aggregation: mean
    higher_is_better: true
metadata:
  version: 3.0
dataset_kwargs:
  trust_remote_code: true
```

cc @haileyschoelkopf 

@Hannibal046 Great work! Would you be willing to make a PR? lets call it `drop_fewshot` maybe, as the original one was taken exactly from the original implementation [see](https://huggingface.co/blog/open-llm-leaderboard-drop#so-whats-next).


> No way this few-shot prompt works...
> 
> ![Image](https://github.com/user-attachments/assets/265eca54-aa94-4247-91f7-5613aaf4835b)

Hi, thanks for the reply! I'd like to make a PR with a new name `drop_fewshot`, however, I believe the original implementation has some potential issue and I think we should fix it first. For example:
1. the doc_to_target function should be: `"{{ answers|join(', ')}}"` rather than `"{{ answer|join(', ')}}"`
2. also, the `get_answers` would omit situation when multiple answers are present.
3. the generate util args with `.` is not correct because many answers in `DROP` dataset are float number

Great THANKS@Hannibal046 !!! I have searched through many respositories and found no one has been able to reproduce llama3-8b on DROP, perhaps you just find out the reason.

> Hi, thanks for the reply! I'd like to make a PR with a new name `drop_fewshot`, however, I believe the original implementation has some potential issue and I think we should fix it first. For example:
> 
> 1. the doc_to_target function should be: `"{{ answers|join(', ')}}"` rather than `"{{ answer|join(', ')}}"`
> 2. also, the `get_answers` would omit situation when multiple answers are present.
> 3. the generate util args with `.` is not correct because many answers in `DROP` dataset are float number

Hi! Thanks for the feedback. There is some good discussion [here](https://github.com/EleutherAI/lm-evaluation-harness/issues/1050) and [here](https://huggingface.co/blog/open-llm-leaderboard-drop). TLDR: While issues were identified, no consensus was reached. I suggest keeping the current implementation as a historical reference and creating `drop_fixed` with documented improvements.

1. True about answers vs answer, though note we use `doc["answers"]` in process_results, so `doc_to_target` isn't currently used.
2. DROP includes cases like "5"/"five"/"5.0" where all are valid. However, we can modify fewshots to use single answers:
```yaml
fewshot_config:
   doc_to_target: {{answers[0][0]}}
```
3. We could use "\n" as the stop token, as they identify in the blog post.

hi there, is there any progress on this?