A complete list of all the LLM evaluation metrics you need to care about!
Tags: llm, ai, chatgpt, opensource
Published: 2024-01-26 06:41:14

Recently, I have been talking to a lot of LLM developers trying to understand the issues they face while building production-grade LLM applications. There's a certain similarity among all those interviews, most of them are not sure what to evaluate beside the extent of hallucinations.

To make that easy for you, here's a compiled list of the most important evaluation metrics you need to consider before launching your LLM application to production. I have also added notebooks for you to try them out:

**Response Quality:**

| Metrics  | Usage | 
|------------|----------|
| [Response Completeness](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/response_quality/completeness.ipynb)        | Evaluate if the response completely resolves the given user query.      | 
| [Response Relevance](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/response_quality/relevance.ipynb)      | Evaluate whether the generated response for the given question, is relevant or not.    | 
| [Response Conciseness](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/response_quality/conciseness.ipynb) | Evaluate how concise the generated response is i.e. the extent of additional irrelevant information in the response.    | 
| [Response Matching ](https://github.com/uptrain-ai/uptrain/blob/main)  | Compare the LLM-generated text with the gold (ideal) response using the defined score metric.  |
| [Response Consistency](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/response_quality/consistency.ipynb)   | Evaluate how consistent the response is with the question asked as well as with the context provided.  |

**Quality of Retrieved Context and Response Groundedness:**

| Metrics  | Usage | 
|------------|----------|
| [Factual Accuracy](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/factual_accuracy.ipynb)       | Evaluate if the facts present in the response can be verified by the retrieved context.    | 
| [Response Completeness wrt Context](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/response_completeness_wrt_context.ipynb)      | Grades how complete the response was for the question specified concerning the information present in the context.| 
| [Context Relevance](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/context_awareness/relevance.ipynb) | Evaluate if the retrieved context contains sufficient information to answer the given question.    | 

**Prompt Security:**

| Metrics  | Usage | 
|------------|----------|
| [Prompt Injection](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/safeguarding/system_prompt_injection.ipynb)       | Identify prompt leakage attacks| 

**Language Quality of Response:**

| Metrics  | Usage | 
|------------|----------|
| [Tone Critique](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/language_features/tone_critique.ipynb)       | Assess if the tone of machine-generated responses matches with the desired persona.| 
| [Language Critique](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/language_features/language_critique.ipynb)       | Evaluate LLM generated responses on multiple aspects - fluence, politeness, grammar, and coherence.| 

**Conversation Quality:**

| Metrics  | Usage | 
|------------|----------|
| [Conversation Satisfaction](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/conversation/conversation_satisfaction.ipynb)       | Measures the user’s satisfaction with the conversation with the AI assistant based on completeness and user acceptance.| 

**Some other Custom Evaluations:**

| Metrics  | Usage | 
|------------|----------|
| [Guideline Adherence](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/custom/guideline_adherence.ipynb)       | Grade how well the LLM adheres to a given custom guideline.| 
| [Custom Prompt Evaluation](https://github.com/uptrain-ai/uptrain/blob/main)       | Evaluate by defining your custom grading prompt.| 
| [Cosine Similarity](https://github.com/uptrain-ai/uptrain/blob/main/examples/checks/custom/cosine_similarity.ipynb)       | Calculate cosine similarity between embeddings of two texts.| 

BTW all these metrics are maintained by [UpTrain](https://github.com/uptrain-ai/uptrain), by far the best open-source tool that I have used for LLM evaluations.