add comparison with GPTs and migration notice; link to web app

README.md

@@ -7,11 +7,19 @@ pipline_tag: text-classficiation
 
 <img src="https://huggingface.co/vectara/hallucination_evaluation_model/resolve/main/candle.png" width="50" height="50" style="display: inline;">  In Loving memory of Simon Mark Hughes...
 
+**Highlights**:
+* HHEM-2.1-Open shows a significant improvement over HHEM-1.0.
+* HHEM-2.1-Open outperforms GPT-3.5-Turbo and even GPT-4.
+* HHEM-2.1-Open can be run on consumer-grade hardware, occupying less than 600MB RAM space at 32-bit precision and elapsing around 1.5 seconds for a 2k-token input on a modern x86 CPU. 
+
+**To HHEM-1.0 users**: HHEM-2.1-Open introduces breaking changes to the usage. Please update your code according to the [new usage](#using-hhem-21-open) below. We are working making it compatible with `transformers.pipeline` and HuggingFace's Inference Endpoint. We apologize for the inconvenience.
 
 HHEM-2.1-Open is a major upgrade to [HHEM-1.0-Open](https://huggingface.co/vectara/hallucination_evaluation_model/tree/hhem-1.0-open) created by [Vectara](https://vectara.com) in November 2023. The HHEM model series are designed for detecting hallucinations in LLMs. They are particularly useful in the context of building retrieval-augmented-generation (RAG) applications where a set of facts is summarized by an LLM, and HHEM can be used to measure the extent to which this summary is factually consistent with the facts.
 
 If you are interested to learn more about RAG or experiment with Vectara, you can [sign up](https://console.vectara.com/signup/?utm_source=huggingface&utm_medium=space&utm_term=hhem-model&utm_content=console&utm_campaign=) for a free Vectara account.
 
+[**Try out HHEM-2.1-Open from your browser without coding** ](http://13.57.203.109:3000/)
+
 ## Hallucination Detection 101
 By "hallucinated" or "factually inconsistent", we mean that a text (hypothesis, to be judged) is not supported by another text (evidence/premise, given). You **always need two** pieces of text to determine whether a text is hallucinated or not. When applied to RAG (retrieval augmented generation), the LLM is provided with several pieces of text (often called facts or context) retrieved from some dataset, and a hallucination would indicate that the summary (hypothesis) is not supported by those facts (evidence). 
 
@@ -20,6 +28,8 @@ For example, given the premise _"The capital of France is Berlin"_, the hypothes
 
 ## Using HHEM-2.1-Open
 
+HHEM-2.1 has some breaking change from HHEM-1.0. Your previous code will not work anymore. While we are working on backward compatibility, please follow the new usage instructions below.
+
 HHEM-2.1-Open can be loaded easily using the `transformers` library. Just remember to set `trust_remote_code=True` to take advantage of the pre-/post-processing code we provided for your convenience. The **input** of the model is a list of pairs of (premise, hypothesis). For each pair, the model will **return** a score between 0 and 1, where 0 means that the hypothesis is not evidenced at all by the premise and 1 means the hypothesis is fully supported by the premise. 
 
 ```python
@@ -44,40 +54,58 @@ model.predict(pairs) # note the predict() method. Do not do model(pairs).
 # tensor([0.0111, 0.6474, 0.1290, 0.8969, 0.1846, 0.0050, 0.0543])
 ```
 
-Note that the order of a pair is important. For example, notice how the 2nd and 3rd examples in the `pairs` list are consistent and hallcuianted, respectively. 
+You may run into a warning message that "Token indices sequence length is longer than the specified maximum sequence length". Please ignore this warning for now. It is a notification inherited from the foundation, T5-base. 
+
+Note that the order of a pair is important. For example, the 2nd and 3rd examples in the `pairs` list are consistent and hallucinated, respectively. 
 
 
 ## HHEM-2.1-Open vs. HHEM-1.0
 
 The major difference between HHEM-2.1-Open and the original HHEM-1.0 is that HHEM-2.1-Open has an unlimited context length, while HHEM-1.0 is capped at 512 tokens. The longer context length allows HHEM-2.1-Open to provide more accurate hallucination detection for RAG which often needs more than 512 tokens. 
 
-The tables below compare the two models on the [AggreFact](https://arxiv.org/pdf/2205.12854) and [RAGTruth](https://arxiv.org/abs/2401.00396) benchmarks. In particualr, on AggreFact, we focus on its SOTA subset (denoted as `AggreFact-SOTA`) which contains summaries generated by Google's T5, Meta's BART, and Google's Pegasus, which are the three latest models in the AggreFact benchmark. The results on RAGTruth's summarization (denoted as `RAGTruth-Summ`) and QA (denoted as `RAGTruth-QA`) subsets are reported separately.
+The tables below compare the two models on the [AggreFact](https://arxiv.org/pdf/2205.12854) and [RAGTruth](https://arxiv.org/abs/2401.00396) benchmarks, as well as GPT-3.5-Turbo and GPT-4. In particular, on AggreFact, we focus on its SOTA subset (denoted as `AggreFact-SOTA`) which contains summaries generated by Google's T5, Meta's BART, and Google's Pegasus, which are the three latest models in the AggreFact benchmark. The results on RAGTruth's summarization (denoted as `RAGTruth-Summ`) and QA (denoted as `RAGTruth-QA`) subsets are reported separately. The GPT-3.5-Turbo and GPT-4 versions are 01-25 and 06-13 respectively. The zero-shot results of the two GPT models were obtained using the prompt template in [this paper](https://arxiv.org/pdf/2303.15621).
 
 Table 1: Performance on AggreFact-SOTA
 | model     |    Balanced Accuracy | F1     | Recall | Precision |
-|:----------------------|---------:|-------:|-------:|----------:|
-| HHEM-1.0              | 0.7887   | 0.9047 | 0.7081 | 0.6728    |
-| HHEM-2.1-Open         | 0.7655   | 0.6677 | 0.6848 | 0.6513    |
+|:------------------------|---------:|-------:|-------:|----------:|
+| HHEM-1.0                | 78.87%   | 90.47% | 70.81% | 67.28%    |
+| HHEM-2.1-Open           | 76.55%   | 66.77% | 68.48% | 65.13%    |
+| GPT-3.5-Turbo zero-shot | 72.19%   | 60.88% | 58.48% | 63.48%    |
+| GPT-4 06-13 zero-shot   | 73.78%   | 63.86% | 53.03% | 80.27%    |
 
 Table 2: Performance on RAGTruth-Summ
 | model     |    Balanced Accuracy | F1         | Recall    | Precision |
 |:----------------------|---------:|-----------:|----------:|----------:|
-| HHEM-1.0              | 0.5336 | 0.1577   | 0.0931 | 0.5135 |
-| HHEM-2.1-Open         | 0.6442 | 0.4883 | 0.3186 | 0.7558 |
+| HHEM-1.0              | 53.36% | 15.77%   | 9.31% | 51.35% |
+| HHEM-2.1-Open         | 64.42% | 44.83% | 31.86% | 75.58% |
+| GPT-3.5-Turbo zero-shot | 58.49%   | 29.72% | 18.14% | 82.22%    |
+| GPT-4 06-13 zero-shot   | 62.62%   | 40.59% | 26.96% | 82.09%    |
 
 Table 3: Performance on RAGTruth-QA
 | model     |    Balanced Accuracy | F1         | Recall    | Precision |
 |:----------------------|---------:|-----------:|----------:|----------:|
-| HHEM-1.0              | 0.5258 | 0.1940   | 0.1625 | 0.2407 |
-| HHEM-2.1-Open         | 0.7428 | 0.6000 | 0.5438 | 0.6692 |
+| HHEM-1.0              | 52.58% | 19.40%   | 16.25% | 24.07% |
+| HHEM-2.1-Open         | 74.28% | 60.00% | 54.38% | 66.92% |
+| GPT-3.5-Turbo zero-shot | 56.16%   | 25.00% | 18.13% | 40.28%    |
+| GPT-4 06-13 zero-shot   | 74.11%   | 57.78% | 56.88% | 58.71%    |
 
-The tables above show that HHEM-2.1-Open has a significant improvement over HHEM-1.0 in the RAGTruth-Summ and RAGTruth-QA benchmarks, while it has a slight decrease in the AggreFact-SOTA benchmark. However when intepreting these results, please note that AggreFact-SOTA is evaluated on relatively older types of LLMs:
+The tables above show that HHEM-2.1-Open has a significant improvement over HHEM-1.0 in the RAGTruth-Summ and RAGTruth-QA benchmarks, while it has a slight decrease in the AggreFact-SOTA benchmark. However, when interpreting these results, please note that AggreFact-SOTA is evaluated on relatively older types of LLMs:
 - LLMs in AggreFact-SOTA: T5, BART, and Pegasus;
 - LLMs in RAGTruth: GPT-4-0613, GPT-3.5-turbo-0613, Llama-2-7B/13B/70B-chat, and Mistral-7B-instruct. 
 
-Therefore, we conclude that HHEM-2.1-Open is better than HHEM-1.0. 
+## HHEM-2.1-Open vs. GPT-3.5-Turbo and GPT-4
+
+From the tables above we can also conclude that HHEM-2.1-Open outperforms both GPT-3.5-Turbo and GPT-4 in all three benchmarks. The quantitative advantage of HHEM-2.1-Open over GPT-3.5-Turbo and GPT-4 is summarized in Table 4 below.
+
+Table 4: Percentage points of HHEM-2.1-Open's balanced accuracies over GPT-3.5-Turbo and GPT-4
+|      |    AggreFact-SOTA | RAGTruth-Summ | RAGTruth-QA |
+|:----------------------|---------:|-----------:|----------:|
+| HHEM-2.1-Open **over** GPT-3.5-Turbo | 4.36% | 5.93% | 18.12% |
+| HHEM-2.1-Open **over** GPT-4         | 2.64% | 1.80% | 0.17% | 
+
+Another advantage of HHEM-2.1-Open is its efficiency. HHEM-2.1-Open can be run on consumer-grade hardware, occupying less than 600MB RAM space at 32-bit precision and elapsing around 1.5 second for a 2k-token input on a modern x86 CPU.
 
-## Want something more powerful?
+## HHEM-2.1: The more powerful, proprietary counterpart of HHEM-2.1-Open
 
 As you may have already sensed from the name, HHEM-2.1-Open is the open source version of the premium HHEM-2.1. HHEM-2.1 (without the `-Open`) is offered exclusively via Vectara's RAG-as-a-service platform. The major difference between HHEM-2.1 and HHEM-2.1-Open is that HHEM-2.1 is cross-lingual on three languages: English, German, and French, while HHEM-2.1-Open is English-only. "Cross-lingual" means any combination of the three languages, e.g., documents in German, query in English, results in French.