Repository: prometheus-eval/prometheus-eval
Issue: Wrong RE Extraction
Root cause: Algorithmic Error

When I running below code
```
from prometheus_eval.vllm import VLLM
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import RELATIVE_PROMPT

model = VLLM(model="prometheus-eval/prometheus-7b-v2.0")
judge = PrometheusEval(model=model, relative_grade_template=RELATIVE_PROMPT)


data = {
  "instruction": "What are the names of some famous actors that started their careers on Broadway?",
  "response_A": "Sure, here's a list of some famous actors that started their careers on Broadway:\n\n* Tony Award-winning actor, Jason Alexander\n* Emmy Award-winning actress, Jennifer Lawrence\n* Olivier Award-winning actor, Patrick Gallagher\n* Grammy Award-winning artist, Jennifer Hudson\n* Oscar-winning actors, Marlon Brando and Denzel Washington",
  "response_B": "Several famous actors started their careers on Broadway before making it big in film and television. Here are a few notable examples:\n\n1. Sarah Jessica Parker - Before she was Carrie Bradshaw on \"Sex and the City,\" Sarah Jessica Parker was a Broadway star, having appeared in productions like \"Annie\" as a child.\n\n2. Meryl Streep - Meryl Streep's early career included Broadway productions such as \"Trelawny of the 'Wells'\" and \"A Memory of Two Mondays / 27 Wagons Full of Cotton.\"\n\n3. Hugh Jackman - Hugh Jackman won a Tony Award for his role in \"The Boy from Oz\" and has been known for his stage work as well as his film career.\n\n4. Sutton Foster - Known for her television role in \"Younger,\" Sutton Foster is also a Broadway legend with leading roles in shows like \"Thoroughly Modern Millie\" and \"Anything Goes.\"\n\n5. Kristen Bell - Before she was the voice of Anna in \"Frozen\" or the star of \"The Good Place,\" Kristen Bell appeared in Broadway's \"The Adventures of Tom Sawyer\" and \"The Crucible.\"\n\n6. Audra McDonald - Audra McDonald is a renowned Broadway actress with a record-breaking number of Tony Awards. She's starred in \"Ragtime,\" \"Carousel,\" \"Master Class,\" and more.\n\n7. Nathan Lane - Nathan Lane is a Broadway veteran known for his roles in \"The Producers,\" \"A Funny Thing Happened on the Way to the Forum,\" and \"Angels in America.\"\n\n8. Idina Menzel - Before \"Frozen\" and \"Wicked\" made her a household name, Idina Menzel started on Broadway in shows like \"Rent\" and \"Hair.\"\n\n9. Lin-Manuel Miranda - Before \"Hamilton\" and \"In the Heights\" became huge hits, Lin-Manuel Miranda was performing on Broadway, eventually becoming a celebrated writer and actor.\n\n10. Lea Michele - Prior to her role on \"Glee,\" Lea Michele was a young Broadway actress in shows like \"Les Misérables,\" \"Ragtime,\" and \"Spring Awakening.\"\n\nThese actors are just a few examples of the many performers who have transitioned from the Broadway stage to broader fame in the entertainment industry. Broadway often serves as a proving ground for talent, and many actors continue to return to the stage throughout their careers.",
  "rubric": ""
}


feedback, score = judge.single_relative_grade(**data)

print("Feedback:", feedback)
print("Score:", score)
```
I get feed back and score like this:
```
Feedback: Response A provides a list of famous actors that started their careers on Broadway, but several of the names mentioned, such as Jennifer Lawrence, Patrick Gallagher, and Marlon Brando, are incorrect. Jason Alexander, while famous, does not start his career on Broadway, and neither does Denzel Washington. Moreover, while Patrick Gallagher is indeed an award-winning actor, he is not particularly well-known outside of theater circles. Hence, the response does not meet the criterion of providing accurate information about actors who began their careers on Broadway.

On the other hand, Response B offers a comprehensive list of actors who began their careers on Broadway and then became famous in film and television. The response correctly identifies Sarah Jessica Parker, Meryl Streep, Hugh Jackman, Sutton Foster, Kristen Bell, Audra McDonald, Nathan Lane, Idina Menzel, Lin-Manuel Miranda, and Lea Michele as actors who started on Broadway. Furthermore, the response provides context for their careers, helping to establish their connection to Broadway.

Given the fact that the actors listed in Response B are indeed known for starting their careers on Broadway and are correctly associated with Broadway, the response aligns with the score rubric, which values accuracy and relevance in answering the question posed. The inclusion of relevant context and examples for each actor enhances the response's quality, making it a more informative and useful answer.

Therefore, based on the score rubric's criteria of accuracy, relevance, and comprehensiveness, Response B is a better response than Response A. 
[RESULT] B
Score: A
```
It is obviously that the score should be B, and I think this is caused by https://github.com/prometheus-eval/prometheus-eval/blob/2d0ce670ee3c251a85d461d4a8a683bf75629010/libs/prometheus-eval/prometheus_eval/parser.py#L32

Actually it has huge influence, I found 40 percent this error in my dataset, I suggest you can revert to this commit https://github.com/prometheus-eval/prometheus-eval/commit/4cc298383adeef12e5c5259a129ac257d99739e7 or you can add an extra mode to control a loose pattern or strict pattern for RE extraction 

Thank you for reporting this issue, @bittersweet1999. I will address it as soon as possible. In the meantime, please use version 0.1.17 as a workaround.


@bittersweet1999 I have modified the parsing function and verified its performance on the pattern mentioned above. However, I will leave this issue open as a thread to track any potential parsing errors that may arise in the future. Thanks again for raising this!