return only harim+ scores

README.md

@@ -4,7 +4,7 @@ emoji: 🤗
 colorFrom: blue
 colorTo: red
 sdk: gradio
-sdk_version: 3.9
+sdk_version: 3.0.2
 app_file: app.py
 pinned: false
 tags:
@@ -34,7 +34,6 @@ pip install evaluate
 import evaluate
 from pprint import pprint
 
-# example from the paper
 art = """Spain's 2-0 defeat by Holland on Tuesday brought back bitter memories of their disastrous 2014 World Cup, but coach Vicente del Bosque will not be too worried about a third straight friendly defeat, insists Gerard Pique. Holland, whose 5-1 drubbing of Spain in the group stage in Brazil last year marked the end of the Iberian nation's six-year domination of the world game, scored two early goals at the Amsterdam Arena and held on against some determined Spain pressure in the second half for a 2-0 success. They became the first team to inflict two defeats on Del Bosque since he took over in 2008 but the gruff 64-year-old had used the match to try out several new faces and he fielded a largely experimental, second-string team. Stefan de Vrij (right) headed Holland in front against Spain at the Amsterdam Arena on Tuesday Gerard Pique (left) could do nothing to stop Davy Klaassen doubling the Dutch advantage Malaga forward Juanmi and Sevilla midfielder Vitolo became the 55th and 56th players to debut under Del Bosque, while the likes of goalkeeper David de Gea, defenders Raul Albiol, Juan Bernat and Dani Carvajal and midfielder Mario Suarez all started the game. 'The national team's state of health is good,' centre back Gerard Pique told reporters. 'We are in a process where players are coming into the team and gathering experience,' added the Barcelona defender. 'We are second in qualifying (for Euro 2016) and these friendly games are for experimenting. 'I am not that worried about this match because we lost friendlies in previous years and then ended up winning titles.' David de Gea was given a start by Vicente del Bosque but could not keep out De Vrij's header here Dani Carvajal (centre) was another squad player given a chance to impress against Holland Del Bosque will be confident he can find the right mix of players to secure Spain's berth at Euro 2016 in France next year, when they will be chasing an unprecedented third straight title. Slovakia are the surprise leaders in qualifying Group C thanks to a 2-1 win over Spain in Zilina in October and have a maximum 15 points from five of 10 matches. Spain are second on 12 points, three ahead of Ukraine, who they beat 1-0 in Seville on Friday. Del Bosque's side host Slovakia in September in a match that could decide who goes through to the finals as group winners. 'The team is in good shape,' forward Pedro told reporters. 'We have a very clear idea of our playing style and we are able to count on people who are gradually making a place for themselves in the team.'"""
 
 summaries = [
@@ -46,8 +45,8 @@ summaries = [
 articles = [art] * len(summaries)
 
 scorer = evaluate.load('NCSOFT/harim_plus')
-scores = scorer.compute(predictions = summaries, references = articles) # use_aggregator=False, tokenwise_score=False, bsz=32)
-pprint(scores['harim+'])
+scores = scorer.compute(predictions = summaries, references = articles) # use_aggregator=False, bsz=32, return_details=False, tokenwise_score=False)
+pprint(scores)
 >>> [1.8230078220367432,
             1.5361897945404053,
             1.806436538696289,

harim_plus.py

@@ -30,14 +30,11 @@ _CITATION = """\
 }
 """
 
-_DESCRIPTION = """\
-HaRiM+ is a reference-less (i.e. scoring summary quality only requires an article) evaluation metric score for summarization task which hurls the power of summarization model.
-It will work great ranking the summary-article pairs according to its quality.
-Note that the score range is unbound.
+_DESCRIPTION = f"""HaRiM+ is a reference-less evaluation metric (i.e. requires only article-summary pair, no reference summary) for summarization which hurls the power of summarization model.
+Summarization model inside the HaRiM+ will read and evaluate how good the quality of a summary given the paired article.
+It will work great for ranking the summary-article pairs according to its quality.
 
-Summarization model inside the HaRiM+ will read and evaluate how good the quality of a summary given the paired source article.
-
-HaRiM+ is proved effective for benchmarking summarization systems (system-level performance) as well as ranking the article-summary pairs (segment-level performance) in comprehensive aspect such as factuality, consistency, coherency, fluency, and relevance. For details, refer to our paper published in AACL2022.
+HaRiM+ is proved effective for benchmarking summarization systems (system-level performance) as well as ranking the article-summary pairs (segment-level performance) in comprehensive aspect such as factuality, consistency, coherency, fluency, and relevance. For details, refer to our [paper]({PAPER_URL}) published in AACL2022.
 """
 
 _KWARGS_DESCRIPTION = """
@@ -51,14 +48,12 @@ Args:
     `predictions` (list of str): generated summaries
     `references` (list of str): source articles to be summarized
     `use_aggregator` (bool): if True, average of the scores are returned
+    `bsz` (int): batch size for harim to iterate through the given pairs
+    `return_details` (bool): whether to show more than harim+ score (returns logppl, harim term. refer to the paper for detail)
+        `tokenwise_score` (bool): whether to show tokenwise scores for input pairs (if return_details=False, this is ignored)
 
 Returns:
-    'results' (dict): {
-        'harim+' (List[float] or float): HaRiM+ score to use,
-        'harim' (List[float] or float): HaRiM term for computing the score above,
-        'log_ppl' (List[float] or float): Log perplexity term. Same as (Yuan et al., NeurIPS 2021),
-        'lambda' (float): (recommend not to modify this) Balancing coeff. for computing harim+ from harim and log_ppl.
-    }
+    'results' (list of float): harim+ score for each summary-article pair
 
 Examples:
     >>> summaries = ["hello there", "hello there"]
@@ -94,8 +89,8 @@ class Harimplus(evaluate.Metric):
             inputs_description=_KWARGS_DESCRIPTION,
             features=datasets.Features(
                 {
-                    "predictions": datasets.Value("string", id="sequence"),
-                    "references": datasets.Value("string", id="sequence"),
+                    "predictions (summaries)": datasets.Value("string", id="sequence"),
+                    "references (articles)": datasets.Value("string", id="sequence"),
                 }
             ),
             codebase_urls=[CODEBASE_URL],
@@ -124,8 +119,9 @@ class Harimplus(evaluate.Metric):
                         references=None,
                         use_aggregator=False,
                         bsz=32,
-                        tokenwise_score=False):
+                        tokenwise_score=False,
+                        return_details=False):
         summaries = predictions
         articles = references
-        scores = self.scorer.compute(predictions=summaries, references=articles,  use_aggregator=use_aggregator, bsz=bsz, tokenwise_score=tokenwise_score)
+        scores = self.scorer.compute(predictions=summaries, references=articles,  use_aggregator=use_aggregator, bsz=bsz, tokenwise_score=tokenwise_score, return_details=return_details)
         return scores