update about page text

app.py

@@ -40,7 +40,7 @@ def update_table(
         "Cost": ["Cost Band"],
         "Accuracy": ["Accuracy", "Instruction Following", "Conciseness", "Completeness", "Factuality"],
         "Speed (Latency)": ["Response Time (Sec)", "Mean Output Tokens"],
-        "Trust & Safety": ["Trust & Safety", "Safety", "Privacy", "Truthfulness", "CRM Bias"],
+        "Trust & Safety": ["Trust & Safety", "Safety", "Privacy", "Truthfulness", "CRM Fairness"],
     }
     all_metric_cols = []
     for area in metric_area_maps:

src/about.py

@@ -8,27 +8,9 @@ INTRODUCTION_TEXT = """
 
 """
 
-# Which evaluations are you running? how can people reproduce what you have?
-# LLM_BENCHMARKS_TEXT = """
-# 1) GPT-4T was used except for some accuracy use cases with atypically long input tokens.
-# 2) Hyperparameters were optimized for a subset of models evaluated (platform models?) Were parameters optimized as well?
-# 3) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
-# 3) Latency reflects the time to receive the entire completion.
-# 4) Some external APIs were direct to the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
-# 5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects
-# 6) All tests on open source models were performed on original models (correct?); custom fine-tuning may impact performance in trust / safety / toxicity / bias / etc.
-# 7) For the tests on latency, the inputs were *approximately* 500 / 3000 tokens. A short prompt was added and different models tokenize differently.
-# 8) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic apis).
-# 9) Something about limitations of LLM judges (despite correlation with human annotators)
-# 10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases)
-# 11) Maybe something about the tasks being primarily summarization / generation
-# 12) CRM T&S is done by perturbing words: 1) for gender bias, we perturb person names and pronouns to opposite gender. 2) for entity bias, we perturb company names to its competitors in the same sector
-# 13) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
-# """
-
 LLM_BENCHMARKS_TEXT = """
-0) We consider models that are general instruction-tuned, not task-specific fine-tuned ones
-1) For GPT-4/GPT-4-Turbo Models, following tasks were evaluated:
+1) We consider models that are general instruction-tuned, not task-specific fine-tuned ones
+2) For GPT-4/GPT-4-Turbo Models, following tasks were evaluated:
   + GPT4:
     - Service: Conversation summary
     - Sales: Email Generation
@@ -43,19 +25,14 @@ LLM_BENCHMARKS_TEXT = """
     - Service: Call Summary
     - Service: Live Chat Summary
 
-2) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
-3) Latency reflects the time to receive the entire completion.
-4) Some external APIs were direct to the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
-5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects
-6) All tests on open source models were performed on original models (correct?); custom fine-tuning may impact performance in trust / safety / toxicity / bias / etc.
-7) For the tests on latency, the inputs were *approximately* 500 / 3000 tokens. A short prompt was added and different models tokenize differently.
-8) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic apis).
-9) Building a reliable LLM-based evaluator remains an open challenge due to inherent biases such as 1) Length Bias: the tendency to favor longer responses, and 2) Self-enhancement Bias: the tendency of the LLM-evaluator to favor its own responses.
-10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases).
-11) Maybe something about the tasks being primarily summarization / generation
-12) Trust & Safety was benchmarked on public datasets as well as bias perturbations on CRM datasets. For gender bias, person names and pronouns were perturbed. For company bias, company names were perturbed to competitors in the same sector. For the CRM Fairness metric, higher means less bias.
-13) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
-14) The current auto-evaluation is based on LLaMA-70B as Judge, which showed the highest correlation with human annotaotors.
+3) Latency scores reflect the mean latency on a high-speed internet connection over a particular time span, based on the time to receive the entire completion; response times for external APIs may vary and be impacted by internet speed, location, etc.
+4) Some external APIs were hosted directly by the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
+5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects.
+6) For the tests on latency: two cases were considered: (1) Length ~500 input and length ~250 output, and (2) length ~3000 input and ~250 output, reflecting common use cases for summarization and generation tasks.
+7) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic APIs).
+8) Trust & Safety was benchmarked on public datasets as well as bias perturbations on CRM datasets. For gender bias, person names and pronouns were perturbed. For company bias, company names were perturbed to competitors in the same sector. For the CRM Fairness metric, higher means less bias.
+9) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
+10) The current auto-evaluation is based on LLaMA-70B as Judge, which showed the highest correlation with human annotators; however, LLM judges may be less reliable than human annotators. This remains an active area of research.
 """
 
 CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"