## UpTrain

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/integrations/callbacks/uptrain.ipynb)

> UpTrain \[[github](https://github.com/uptrain-ai/uptrain) || [website](https://uptrain.ai/) || [docs](https://docs.uptrain.ai/getting-started/introduction)\] is an open-source platform to evaluate and improve LLM applications. It provides grades for 20+ preconfigured checks (covering language, code, embedding use cases), performs root cause analyses on instances of failure cases and provides guidance for resolving them.

## UpTrain Callback Handler[​](#uptrain-callback-handler "Direct link to UpTrain Callback Handler")

This notebook showcases the UpTrain callback handler seamlessly integrating into your pipeline, facilitating diverse evaluations. We have chosen a few evaluations that we deemed apt for evaluating the chains. These evaluations run automatically, with results displayed in the output. More details on UpTrain’s evaluations can be found [here](https://github.com/uptrain-ai/uptrain?tab=readme-ov-file#pre-built-evaluations-we-offer-).

Selected retievers from Langchain are highlighted for demonstration:

### 1\. **Vanilla RAG**:[​](#vanilla-rag "Direct link to vanilla-rag")

RAG plays a crucial role in retrieving context and generating responses. To ensure its performance and response quality, we conduct the following evaluations:

*   **[Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance)**: Determines if the context extracted from the query is relevant to the response.
*   **[Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy)**: Assesses if the LLM is hallcuinating or providing incorrect information.
*   **[Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness)**: Checks if the response contains all the information requested by the query.

### 2\. **Multi Query Generation**:[​](#multi-query-generation "Direct link to multi-query-generation")

MultiQueryRetriever creates multiple variants of a question having a similar meaning to the original question. Given the complexity, we include the previous evaluations and add:

*   **[Multi Query Accuracy](https://docs.uptrain.ai/predefined-evaluations/query-quality/multi-query-accuracy)**: Assures that the multi-queries generated mean the same as the original query.

### 3\. **Context Compression and Reranking**:[​](#context-compression-and-reranking "Direct link to context-compression-and-reranking")

Re-ranking involves reordering nodes based on relevance to the query and choosing top n nodes. Since the number of nodes can reduce once the re-ranking is complete, we perform the following evaluations:

*   **[Context Reranking](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-reranking)**: Checks if the order of re-ranked nodes is more relevant to the query than the original order.
*   **[Context Conciseness](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-conciseness)**: Examines whether the reduced number of nodes still provides all the required information.

These evaluations collectively ensure the robustness and effectiveness of the RAG, MultiQueryRetriever, and the Reranking process in the chain.

## Install Dependencies[​](#install-dependencies "Direct link to Install Dependencies")

```
%pip install -qU langchain langchain_openai uptrain faiss-cpu flashrank
```

```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...To disable this warning, you can either:    - Avoid using `tokenizers` before the fork if possible    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
```

```
WARNING: There was an error checking the latest version of pip.Note: you may need to restart the kernel to use updated packages.
```

NOTE: that you can also install `faiss-gpu` instead of `faiss-cpu` if you want to use the GPU enabled version of the library.

## Import Libraries[​](#import-libraries "Direct link to Import Libraries")

```
from getpass import getpassfrom langchain.chains import RetrievalQAfrom langchain.retrievers import ContextualCompressionRetrieverfrom langchain.retrievers.document_compressors import FlashrankRerankfrom langchain.retrievers.multi_query import MultiQueryRetrieverfrom langchain_community.callbacks.uptrain_callback import UpTrainCallbackHandlerfrom langchain_community.document_loaders import TextLoaderfrom langchain_community.vectorstores import FAISSfrom langchain_core.output_parsers.string import StrOutputParserfrom langchain_core.prompts.chat import ChatPromptTemplatefrom langchain_core.runnables.passthrough import RunnablePassthroughfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain_text_splitters import (    RecursiveCharacterTextSplitter,)
```

## Load the documents[​](#load-the-documents "Direct link to Load the documents")

```
loader = TextLoader("../../modules/state_of_the_union.txt")documents = loader.load()
```

## Split the document into chunks[​](#split-the-document-into-chunks "Direct link to Split the document into chunks")

```
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)chunks = text_splitter.split_documents(documents)
```

## Create the retriever[​](#create-the-retriever "Direct link to Create the retriever")

```
embeddings = OpenAIEmbeddings()db = FAISS.from_documents(chunks, embeddings)retriever = db.as_retriever()
```

## Define the LLM[​](#define-the-llm "Direct link to Define the LLM")

```
llm = ChatOpenAI(temperature=0, model="gpt-4")
```

## Set the openai API key[​](#set-the-openai-api-key "Direct link to Set the openai API key")

This key is required to perform the evaluations. UpTrain uses the GPT models to evaluate the responses generated by the LLM.

```
OPENAI_API_KEY = getpass()
```

## Setup[​](#setup "Direct link to Setup")

For each of the retrievers below, it is better to define the callback handler again to avoid interference. You can choose between the following options for evaluating using UpTrain:

### 1\. **UpTrain’s Open-Source Software (OSS)**:[​](#uptrains-open-source-software-oss "Direct link to uptrains-open-source-software-oss")

You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. You can get yours [here](https://platform.openai.com/account/api-keys).

Parameters: - key\_type=“openai” - api\_key=“OPENAI\_API\_KEY” - project\_name\_prefix=“PROJECT\_NAME\_PREFIX”

### 2\. **UpTrain Managed Service and Dashboards**:[​](#uptrain-managed-service-and-dashboards "Direct link to uptrain-managed-service-and-dashboards")

You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).

UpTrain Managed service provides: 1. Dashboards with advanced drill-down and filtering options 1. Insights and common topics among failing cases 1. Observability and real-time monitoring of production data 1. Regression testing via seamless integration with your CI/CD pipelines

The notebook contains some screenshots of the dashboards and the insights that you can get from the UpTrain managed service.

Parameters: - key\_type=“uptrain” - api\_key=“UPTRAIN\_API\_KEY” - project\_name\_prefix=“PROJECT\_NAME\_PREFIX”

**Note:** The `project_name_prefix` will be used as prefix for the project names in the UpTrain dashboard. These will be different for different types of evals. For example, if you set project\_name\_prefix=“langchain” and perform the multi\_query evaluation, the project name will be “langchain\_multi\_query”.

## 1\. Vanilla RAG

UpTrain callback handler will automatically capture the query, context and response once generated and will run the following three evaluations _(Graded from 0 to 1)_ on the response: - **[Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance)**: Check if the context extractedfrom the query is relevant to the response. - **[Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy)**: Check how factually accurate the response is. - **[Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness)**: Check if the response contains all the information that the query is asking for.

```
# Create the RAG prompttemplate = """Answer the question based only on the following context, which can include text and tables:{context}Question: {question}"""rag_prompt_text = ChatPromptTemplate.from_template(template)# Create the chainchain = (    {"context": retriever, "question": RunnablePassthrough()}    | rag_prompt_text    | llm    | StrOutputParser())# Create the uptrain callback handleruptrain_callback = UpTrainCallbackHandler(key_type="openai", api_key=OPENAI_API_KEY)config = {"callbacks": [uptrain_callback]}# Invoke the chain with a queryquery = "What did the president say about Ketanji Brown Jackson"docs = chain.invoke(query, config=config)
```

```
2024-04-17 17:03:44.969 | INFO     | uptrain.framework.evalllm:evaluate_on_server:378 - Sending evaluation request for rows 0 to <50 to the Uptrain2024-04-17 17:04:05.809 | INFO     | uptrain.framework.evalllm:evaluate:367 - Local server not running, start the server to log data and visualize in the dashboard!
```

```
Question: What did the president say about Ketanji Brown JacksonResponse: The president mentioned that he had nominated Ketanji Brown Jackson to serve on the United States Supreme Court 4 days ago. He described her as one of the nation's top legal minds who will continue Justice Breyer’s legacy of excellence. He also mentioned that she is a former top litigator in private practice, a former federal public defender, and comes from a family of public school educators and police officers. He described her as a consensus builder and noted that since her nomination, she has received a broad range of support from various groups, including the Fraternal Order of Police and former judges appointed by both Democrats and Republicans.Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 1.0
```

## 2\. Multi Query Generation

The **MultiQueryRetriever** is used to tackle the problem that the RAG pipeline might not return the best set of documents based on the query. It generates multiple queries that mean the same as the original query and then fetches documents for each.

To evluate this retriever, UpTrain will run the following evaluation: - **[Multi Query Accuracy](https://docs.uptrain.ai/predefined-evaluations/query-quality/multi-query-accuracy)**: Checks if the multi-queries generated mean the same as the original query.

```
# Create the retrievermulti_query_retriever = MultiQueryRetriever.from_llm(retriever=retriever, llm=llm)# Create the uptrain callbackuptrain_callback = UpTrainCallbackHandler(key_type="openai", api_key=OPENAI_API_KEY)config = {"callbacks": [uptrain_callback]}# Create the RAG prompttemplate = """Answer the question based only on the following context, which can include text and tables:{context}Question: {question}"""rag_prompt_text = ChatPromptTemplate.from_template(template)chain = (    {"context": multi_query_retriever, "question": RunnablePassthrough()}    | rag_prompt_text    | llm    | StrOutputParser())# Invoke the chain with a queryquestion = "What did the president say about Ketanji Brown Jackson"docs = chain.invoke(question, config=config)
```

```
2024-04-17 17:04:10.675 | INFO     | uptrain.framework.evalllm:evaluate_on_server:378 - Sending evaluation request for rows 0 to <50 to the Uptrain2024-04-17 17:04:16.804 | INFO     | uptrain.framework.evalllm:evaluate:367 - Local server not running, start the server to log data and visualize in the dashboard!2024-04-17 17:04:22.027 | INFO     | uptrain.framework.evalllm:evaluate_on_server:378 - Sending evaluation request for rows 0 to <50 to the Uptrain2024-04-17 17:04:44.033 | INFO     | uptrain.framework.evalllm:evaluate:367 - Local server not running, start the server to log data and visualize in the dashboard!
```

```
Question: What did the president say about Ketanji Brown JacksonMulti Queries:  - How did the president comment on Ketanji Brown Jackson?  - What were the president's remarks regarding Ketanji Brown Jackson?  - What statements has the president made about Ketanji Brown Jackson?Multi Query Accuracy Score: 0.5Question: What did the president say about Ketanji Brown JacksonResponse: The president mentioned that he had nominated Circuit Court of Appeals Judge Ketanji Brown Jackson to serve on the United States Supreme Court 4 days ago. He described her as one of the nation's top legal minds who will continue Justice Breyer’s legacy of excellence. He also mentioned that since her nomination, she has received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 1.0
```

## 3\. Context Compression and Reranking

The reranking process involves reordering nodes based on relevance to the query and choosing the top n nodes. Since the number of nodes can reduce once the reranking is complete, we perform the following evaluations: - **[Context Reranking](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-reranking)**: Check if the order of re-ranked nodes is more relevant to the query than the original order. - **[Context Conciseness](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-conciseness)**: Check if the reduced number of nodes still provides all the required information.

```
# Create the retrievercompressor = FlashrankRerank()compression_retriever = ContextualCompressionRetriever(    base_compressor=compressor, base_retriever=retriever)# Create the chainchain = RetrievalQA.from_chain_type(llm=llm, retriever=compression_retriever)# Create the uptrain callbackuptrain_callback = UpTrainCallbackHandler(key_type="openai", api_key=OPENAI_API_KEY)config = {"callbacks": [uptrain_callback]}# Invoke the chain with a queryquery = "What did the president say about Ketanji Brown Jackson"result = chain.invoke(query, config=config)
```

```
2024-04-17 17:04:46.462 | INFO     | uptrain.framework.evalllm:evaluate_on_server:378 - Sending evaluation request for rows 0 to <50 to the Uptrain2024-04-17 17:04:53.561 | INFO     | uptrain.framework.evalllm:evaluate:367 - Local server not running, start the server to log data and visualize in the dashboard!2024-04-17 17:04:56.947 | INFO     | uptrain.framework.evalllm:evaluate_on_server:378 - Sending evaluation request for rows 0 to <50 to the Uptrain2024-04-17 17:05:16.551 | INFO     | uptrain.framework.evalllm:evaluate:367 - Local server not running, start the server to log data and visualize in the dashboard!
```

```
Question: What did the president say about Ketanji Brown JacksonContext Conciseness Score: 0.0Context Reranking Score: 1.0Question: What did the president say about Ketanji Brown JacksonResponse: The President mentioned that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson to serve on the United States Supreme Court 4 days ago. He described her as one of the nation's top legal minds who will continue Justice Breyer’s legacy of excellence.Context Relevance Score: 1.0Factual Accuracy Score: 1.0Response Completeness Score: 0.5
```