extract entailment_checker

README.md

@@ -27,6 +27,8 @@ license: apache-2.0
     - [Limits and possible improvements](#limits-and-possible-improvements)
     - [Repository structure](#repository-structure)
     - [Installation](#installation)
+      - [Entailment Checker node](#entailment-checker-node)
+      - [Fact Checking 🎸 Rocks!](#fact-checking--rocks)
 
 ### Idea
 💡 This project aims to show that a *naive and simple baseline* for fact checking can be built by combining dense retrieval and a textual entailment task.
@@ -42,7 +44,7 @@ In a nutshell, the flow is as follows:
 - [🧑‍🏫 Slides](./presentation/fact_checking_rocks.pdf)
 
 ### System description
-🪄 This project is strongly based on [🔎 Haystack](https://github.com/deepset-ai/haystack), an open source NLP framework to realize search system. The main components of our system are an indexing pipeline and a search pipeline.
+🪄 This project is strongly based on [🔎 Haystack](https://github.com/deepset-ai/haystack), an open source NLP framework that enables seamless use of Transformer models and LLMs to interact with your data. The main components of our system are an indexing pipeline and a search pipeline.
 
 #### Indexing pipeline
 * [Crawling](https://github.com/anakin87/fact-checking-rocks/blob/321ba7893bbe79582f8c052493acfda497c5b785/notebooks/get_wikipedia_data.ipynb): Crawl data from Wikipedia, starting from the page [List of mainstream rock performers](https://en.wikipedia.org/wiki/List_of_mainstream_rock_performers) and using the [python wrapper](https://github.com/goldsmith/Wikipedia)
@@ -58,7 +60,8 @@ In a nutshell, the flow is as follows:
 * the user enters a factual statement
 * compute the embedding of the user statement using the same Sentence Transformer used for indexing (`msmarco-distilbert-base-tas-b`)
 * retrieve the K most relevant text passages stored in FAISS (along with their relevance scores)
-* **text entailment task**: compute the text entailment between each text passage (premise) and the user statement (hypotesis), using a Natural Language Inference model (`microsoft/deberta-v2-xlarge-mnli`). For every text passage, we have 3 scores (summing to 1): entailment, contradiction and neutral. *(For this task, I developed a custom Haystack node: `EntailmentChecker`)*
+* the following steps are performed using the [`EntailmentChecker`, a custom Haystack node](https://github.com/anakin87/haystack-entailment-checker)
+* **text entailment task**: compute the text entailment between each text passage (premise) and the user statement (hypothesis), using a Natural Language Inference model (`microsoft/deberta-v2-xlarge-mnli`). For every text passage, we have 3 scores (summing to 1): entailment, contradiction and neutral.
 * aggregate the text entailment scores: compute the weighted average of them, where the weight is the relevance score. **Now it is possible to tell if the knowledge base confirms, is neutral or disproves the user statement.**
 * *empirical consideration: if in the first N passages (N<K),  there is strong evidence of entailment/contradiction (partial aggregate scores > 0.5), it is better not to consider (K-N) less relevant documents.*
 
@@ -83,7 +86,15 @@ While keeping this simple approach, some **improvements** could be made:
 * [data folder](./data/): all necessary data, including original Wikipedia data, FAISS Index and prepared random statements
 
 ### Installation
-💻 To install this project locally, follow these steps:
+💻
+#### Entailment Checker node
+If you want to build a similar system using the [`EntailmentChecker`](https://github.com/anakin87/haystack-entailment-checker), I strongly suggest taking a look at [the node repository](https://github.com/anakin87/haystack-entailment-checker). It can be easily installed with
+```bash
+pip install haystack-entailment-checker
+```
+
+#### Fact Checking 🎸 Rocks!
+ To install this project locally, follow these steps:
 * `git clone https://github.com/anakin87/fact-checking-rocks`
 * `cd fact-checking-rocks`
 * `pip install -r requirements.txt`

app_utils/backend_utils.py

@@ -7,7 +7,7 @@ from haystack.nodes import EmbeddingRetriever, PromptNode
 from haystack.pipelines import Pipeline
 import streamlit as st
 
-from app_utils.entailment_checker import EntailmentChecker
+from haystack_entailment_checker import EntailmentChecker
 from app_utils.config import (
     STATEMENTS_PATH,
     INDEX_DIR,

app_utils/entailment_checker.py

@@ -1,126 +0,0 @@
-from typing import List, Optional
-
-from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoConfig
-import torch
-from haystack.nodes.base import BaseComponent
-from haystack.modeling.utils import initialize_device_settings
-from haystack.schema import Document
-
-
-class EntailmentChecker(BaseComponent):
-    """
-    This node checks the entailment between every document content and the query.
-    It enrichs the documents metadata with entailment informations.
-    It also returns aggregate entailment information.
-    """
-
-    outgoing_edges = 1
-
-    def __init__(
-        self,
-        model_name_or_path: str = "roberta-large-mnli",
-        model_version: Optional[str] = None,
-        tokenizer: Optional[str] = None,
-        use_gpu: bool = True,
-        batch_size: int = 16,
-        entailment_contradiction_threshold: float = 0.5,
-    ):
-        """
-        Load a Natural Language Inference model from Transformers.
-
-        :param model_name_or_path: Directory of a saved model or the name of a public model.
-        See https://huggingface.co/models for full list of available models.
-        :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.
-        :param tokenizer: Name of the tokenizer (usually the same as model)
-        :param use_gpu: Whether to use GPU (if available).
-        :param batch_size: Number of Documents to be processed at a time.
-        :param entailment_contradiction_threshold: if in the first N documents there is a strong evidence of entailment/contradiction
-        (aggregate entailment or contradiction are greater than the threshold), the less relevant documents are not taken into account
-        """
-        super().__init__()
-
-        self.devices, _ = initialize_device_settings(use_cuda=use_gpu, multi_gpu=False)
-
-        tokenizer = tokenizer or model_name_or_path
-        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer)
-        self.model = AutoModelForSequenceClassification.from_pretrained(
-            pretrained_model_name_or_path=model_name_or_path, revision=model_version
-        )
-        self.batch_size = batch_size
-        self.entailment_contradiction_threshold = entailment_contradiction_threshold
-        self.model.to(str(self.devices[0]))
-
-        id2label = AutoConfig.from_pretrained(model_name_or_path).id2label
-        self.labels = [id2label[k].lower() for k in sorted(id2label)]
-        if "entailment" not in self.labels:
-            raise ValueError(
-                "The model config must contain entailment value in the id2label dict."
-            )
-
-    def run(self, query: str, documents: List[Document]):
-
-        scores, agg_con, agg_neu, agg_ent = 0, 0, 0, 0
-        premise_batch = [doc.content for doc in documents]
-        hypotesis_batch = [query] * len(documents)
-        entailment_info_batch = self.get_entailment_batch(premise_batch=premise_batch, hypotesis_batch=hypotesis_batch)
-        for i, (doc, entailment_info) in enumerate(zip(documents, entailment_info_batch)):
-            doc.meta["entailment_info"] = entailment_info
-
-            scores += doc.score
-            con, neu, ent = (
-                entailment_info["contradiction"],
-                entailment_info["neutral"],
-                entailment_info["entailment"],
-            )
-            agg_con += con * doc.score
-            agg_neu += neu * doc.score
-            agg_ent += ent * doc.score
-
-            # if in the first documents there is a strong evidence of entailment/contradiction,
-            # there is no need to consider less relevant documents
-            if max(agg_con, agg_ent) / scores > self.entailment_contradiction_threshold:
-                break
-
-        aggregate_entailment_info = {
-            "contradiction": round(agg_con / scores, 2),
-            "neutral": round(agg_neu / scores, 2),
-            "entailment": round(agg_ent / scores, 2),
-        }
-
-        entailment_checker_result = {
-            "documents": documents[: i + 1],
-            "aggregate_entailment_info": aggregate_entailment_info,
-        }
-
-        return entailment_checker_result, "output_1"
-
-    def run_batch(self, queries: List[str], documents: List[Document]):
-        entailment_checker_result_batch = []
-        entailment_info_batch = self.get_entailment_batch(premise_batch=documents, hypotesis_batch=queries)
-        for doc, entailment_info in zip(documents, entailment_info_batch):
-            doc.meta["entailment_info"] = entailment_info
-            aggregate_entailment_info = {
-                "contradiction": round(entailment_info["contradiction"] / doc.score),
-                "neutral": round(entailment_info["neutral"] / doc.score),
-                "entailment": round(entailment_info["entailment"] / doc.score),
-            }
-            entailment_checker_result_batch.append({
-                "documents": [doc],
-                "aggregate_entailment_info": aggregate_entailment_info,
-            })
-        return entailment_checker_result_batch, "output_1"
-
-
-    def get_entailment_dict(self, probs):
-        entailment_dict = {k.lower(): v for k, v in zip(self.labels, probs)}
-        return entailment_dict
-
-    def get_entailment_batch(self, premise_batch: List[str], hypotesis_batch: List[str]):
-        formatted_texts = [f"{premise}{self.tokenizer.sep_token}{hypotesis}" for premise, hypotesis in zip(premise_batch, hypotesis_batch)]
-        with torch.inference_mode():
-            inputs = self.tokenizer(formatted_texts, return_tensors="pt", padding=True, truncation=True).to(self.devices[0])
-            out = self.model(**inputs)
-            logits = out.logits
-            probs_batch = (torch.nn.functional.softmax(logits, dim=-1).detach().cpu().numpy() )
-        return [self.get_entailment_dict(probs) for probs in probs_batch]
-

requirements.txt

@@ -1,4 +1,5 @@
-farm-haystack[faiss]==1.16.1
+farm-haystack[faiss,inference]==1.18.1
+haystack-entailment-checker
 plotly==5.14.1
 
 # commented to not interfere with streamlit SDK in HF spces