*   [](/v0.2/)
*   [Tutorials](/v0.2/docs/tutorials/)
*   Conversational RAG

On this page

Conversational RAG
==================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Chat history](/v0.2/docs/concepts/#chat-history)
*   [Chat models](/v0.2/docs/concepts/#chat-models)
*   [Embeddings](/v0.2/docs/concepts/#embedding-models)
*   [Vector stores](/v0.2/docs/concepts/#vector-stores)
*   [Retrieval-augmented generation](/v0.2/docs/tutorials/rag/)
*   [Tools](/v0.2/docs/concepts/#tools)
*   [Agents](/v0.2/docs/concepts/#agents)

In many Q&A applications we want to allow the user to have a back-and-forth conversation, meaning the application needs some sort of "memory" of past questions and answers, and some logic for incorporating those into its current thinking.

In this guide we focus on **adding logic for incorporating historical messages.** Further details on chat history management is [covered here](/v0.2/docs/how_to/message_history/).

We will cover two approaches:

1.  Chains, in which we always execute a retrieval step;
2.  Agents, in which we give an LLM discretion over whether and how to execute a retrieval step (or multiple steps).

For the external knowledge source, we will use the same [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng from the [RAG tutorial](/v0.2/docs/tutorials/rag/).

Setup[​](#setup "Direct link to Setup")
---------------------------------------

### Dependencies[​](#dependencies "Direct link to Dependencies")

We'll use OpenAI embeddings and a Chroma vector store in this walkthrough, but everything shown here works with any [Embeddings](/v0.2/docs/concepts/#embedding-models), and [VectorStore](/v0.2/docs/concepts/#vectorstores) or [Retriever](/v0.2/docs/concepts/#retrievers).

We'll use the following packages:

    %%capture --no-stderr%pip install --upgrade --quiet  langchain langchain-community langchainhub langchain-chroma bs4

We need to set environment variable `OPENAI_API_KEY`, which can be done directly or loaded from a `.env` file like so:

    import getpassimport osif not os.environ.get("OPENAI_API_KEY"):    os.environ["OPENAI_API_KEY"] = getpass.getpass()# import dotenv# dotenv.load_dotenv()

### LangSmith[​](#langsmith "Direct link to LangSmith")

Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with [LangSmith](https://smith.langchain.com).

Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:

    os.environ["LANGCHAIN_TRACING_V2"] = "true"if not os.environ.get("LANGCHAIN_API_KEY"):    os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()

Chains[​](#chains "Direct link to Chains")
------------------------------------------

Let's first revisit the Q&A app we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [RAG tutorial](/v0.2/docs/tutorials/rag/).

*   OpenAI
*   Anthropic
*   Azure
*   Google
*   Cohere
*   FireworksAI
*   Groq
*   MistralAI
*   TogetherAI

    pip install -qU langchain-openai

    import getpassimport osos.environ["OPENAI_API_KEY"] = getpass.getpass()from langchain_openai import ChatOpenAIllm = ChatOpenAI(model="gpt-3.5-turbo-0125")

    pip install -qU langchain-anthropic

    import getpassimport osos.environ["ANTHROPIC_API_KEY"] = getpass.getpass()from langchain_anthropic import ChatAnthropicllm = ChatAnthropic(model="claude-3-sonnet-20240229")

    pip install -qU langchain-openai

    import getpassimport osos.environ["AZURE_OPENAI_API_KEY"] = getpass.getpass()from langchain_openai import AzureChatOpenAIllm = AzureChatOpenAI(    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],    azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"],    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],)

    pip install -qU langchain-google-vertexai

    import getpassimport osos.environ["GOOGLE_API_KEY"] = getpass.getpass()from langchain_google_vertexai import ChatVertexAIllm = ChatVertexAI(model="gemini-pro")

    pip install -qU langchain-cohere

    import getpassimport osos.environ["COHERE_API_KEY"] = getpass.getpass()from langchain_cohere import ChatCoherellm = ChatCohere(model="command-r")

    pip install -qU langchain-fireworks

    import getpassimport osos.environ["FIREWORKS_API_KEY"] = getpass.getpass()from langchain_fireworks import ChatFireworksllm = ChatFireworks(model="accounts/fireworks/models/mixtral-8x7b-instruct")

    pip install -qU langchain-groq

    import getpassimport osos.environ["GROQ_API_KEY"] = getpass.getpass()from langchain_groq import ChatGroqllm = ChatGroq(model="llama3-8b-8192")

    pip install -qU langchain-mistralai

    import getpassimport osos.environ["MISTRAL_API_KEY"] = getpass.getpass()from langchain_mistralai import ChatMistralAIllm = ChatMistralAI(model="mistral-large-latest")

    pip install -qU langchain-openai

    import getpassimport osos.environ["TOGETHER_API_KEY"] = getpass.getpass()from langchain_openai import ChatOpenAIllm = ChatOpenAI(    base_url="https://api.together.xyz/v1",    api_key=os.environ["TOGETHER_API_KEY"],    model="mistralai/Mixtral-8x7B-Instruct-v0.1",)

    import bs4from langchain import hubfrom langchain.chains import create_retrieval_chainfrom langchain.chains.combine_documents import create_stuff_documents_chainfrom langchain_chroma import Chromafrom langchain_community.document_loaders import WebBaseLoaderfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import OpenAIEmbeddingsfrom langchain_text_splitters import RecursiveCharacterTextSplitter# 1. Load, chunk and index the contents of the blog to create a retriever.loader = WebBaseLoader(    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),    bs_kwargs=dict(        parse_only=bs4.SoupStrainer(            class_=("post-content", "post-title", "post-header")        )    ),)docs = loader.load()text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)splits = text_splitter.split_documents(docs)vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())retriever = vectorstore.as_retriever()# 2. Incorporate the retriever into a question-answering chain.system_prompt = (    "You are an assistant for question-answering tasks. "    "Use the following pieces of retrieved context to answer "    "the question. If you don't know the answer, say that you "    "don't know. Use three sentences maximum and keep the "    "answer concise."    "\n\n"    "{context}")prompt = ChatPromptTemplate.from_messages(    [        ("system", system_prompt),        ("human", "{input}"),    ])question_answer_chain = create_stuff_documents_chain(llm, prompt)rag_chain = create_retrieval_chain(retriever, question_answer_chain)

**API Reference:**[create\_retrieval\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.retrieval.create_retrieval_chain.html) | [create\_stuff\_documents\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html) | [WebBaseLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) | [ChatPromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.ChatPromptTemplate.html) | [OpenAIEmbeddings](https://api.python.langchain.com/en/latest/embeddings/langchain_openai.embeddings.base.OpenAIEmbeddings.html) | [RecursiveCharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html)

    response = rag_chain.invoke({"input": "What is Task Decomposition?"})response["answer"]

    "Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model. This process helps in guiding the agent through the various subgoals required to achieve the overall task efficiently. Different techniques like Chain of Thought and Tree of Thoughts can be used to decompose tasks into step-by-step processes, enhancing performance and understanding of the model's thinking process."

Note that we have used the built-in chain constructors `create_stuff_documents_chain` and `create_retrieval_chain`, so that the basic ingredients to our solution are:

1.  retriever;
2.  prompt;
3.  LLM.

This will simplify the process of incorporating chat history.

### Adding chat history[​](#adding-chat-history "Direct link to Adding chat history")

The chain we have built uses the input query directly to retrieve relevant context. But in a conversational setting, the user query might require conversational context to be understood. For example, consider this exchange:

> Human: "What is Task Decomposition?"
> 
> AI: "Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model."
> 
> Human: "What are common ways of doing it?"

In order to answer the second question, our system needs to understand that "it" refers to "Task Decomposition."

We'll need to update two things about our existing app:

1.  **Prompt**: Update our prompt to support historical messages as an input.
2.  **Contextualizing questions**: Add a sub-chain that takes the latest user question and reformulates it in the context of the chat history. This can be thought of simply as building a new "history aware" retriever. Whereas before we had:
    *   `query` -> `retriever`  
        Now we will have:
    *   `(query, conversation history)` -> `LLM` -> `rephrased query` -> `retriever`

#### Contextualizing the question[​](#contextualizing-the-question "Direct link to Contextualizing the question")

First we'll need to define a sub-chain that takes historical messages and the latest user question, and reformulates the question if it makes reference to any information in the historical information.

We'll use a prompt that includes a `MessagesPlaceholder` variable under the name "chat\_history". This allows us to pass in a list of Messages to the prompt using the "chat\_history" input key, and these messages will be inserted after the system message and before the human message containing the latest question.

Note that we leverage a helper function [create\_history\_aware\_retriever](https://api.python.langchain.com/en/latest/chains/langchain.chains.history_aware_retriever.create_history_aware_retriever.html) for this step, which manages the case where `chat_history` is empty, and otherwise applies `prompt | llm | StrOutputParser() | retriever` in sequence.

`create_history_aware_retriever` constructs a chain that accepts keys `input` and `chat_history` as input, and has the same output schema as a retriever.

    from langchain.chains import create_history_aware_retrieverfrom langchain_core.prompts import MessagesPlaceholdercontextualize_q_system_prompt = (    "Given a chat history and the latest user question "    "which might reference context in the chat history, "    "formulate a standalone question which can be understood "    "without the chat history. Do NOT answer the question, "    "just reformulate it if needed and otherwise return it as is.")contextualize_q_prompt = ChatPromptTemplate.from_messages(    [        ("system", contextualize_q_system_prompt),        MessagesPlaceholder("chat_history"),        ("human", "{input}"),    ])history_aware_retriever = create_history_aware_retriever(    llm, retriever, contextualize_q_prompt)

**API Reference:**[create\_history\_aware\_retriever](https://api.python.langchain.com/en/latest/chains/langchain.chains.history_aware_retriever.create_history_aware_retriever.html) | [MessagesPlaceholder](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.MessagesPlaceholder.html)

This chain prepends a rephrasing of the input query to our retriever, so that the retrieval incorporates the context of the conversation.

Now we can build our full QA chain. This is as simple as updating the retriever to be our new `history_aware_retriever`.

Again, we will use [create\_stuff\_documents\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html) to generate a `question_answer_chain`, with input keys `context`, `chat_history`, and `input`\-- it accepts the retrieved context alongside the conversation history and query to generate an answer. A more detailed explaination is over [here](/v0.2/docs/tutorials/rag/#built-in-chains)

We build our final `rag_chain` with [create\_retrieval\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.retrieval.create_retrieval_chain.html). This chain applies the `history_aware_retriever` and `question_answer_chain` in sequence, retaining intermediate outputs such as the retrieved context for convenience. It has input keys `input` and `chat_history`, and includes `input`, `chat_history`, `context`, and `answer` in its output.

    from langchain.chains import create_retrieval_chainfrom langchain.chains.combine_documents import create_stuff_documents_chainqa_prompt = ChatPromptTemplate.from_messages(    [        ("system", system_prompt),        MessagesPlaceholder("chat_history"),        ("human", "{input}"),    ])question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

**API Reference:**[create\_retrieval\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.retrieval.create_retrieval_chain.html) | [create\_stuff\_documents\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html)

Let's try this. Below we ask a question and a follow-up question that requires contextualization to return a sensible response. Because our chain includes a `"chat_history"` input, the caller needs to manage the chat history. We can achieve this by appending input and output messages to a list:

    from langchain_core.messages import AIMessage, HumanMessagechat_history = []question = "What is Task Decomposition?"ai_msg_1 = rag_chain.invoke({"input": question, "chat_history": chat_history})chat_history.extend(    [        HumanMessage(content=question),        AIMessage(content=ai_msg_1["answer"]),    ])second_question = "What are common ways of doing it?"ai_msg_2 = rag_chain.invoke({"input": second_question, "chat_history": chat_history})print(ai_msg_2["answer"])

**API Reference:**[AIMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html) | [HumanMessage](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.human.HumanMessage.html)

    Task decomposition can be achieved through various methods such as using techniques like Chain of Thought (CoT) or Tree of Thoughts to break down complex tasks into smaller steps. Common ways include prompting the model with simple instructions like "Steps for XYZ" or task-specific instructions like "Write a story outline." Human inputs can also be used to guide the task decomposition process effectively.

tip

Check out the [LangSmith trace](https://smith.langchain.com/public/243301e4-4cc5-4e52-a6e7-8cfe9208398d/r)

#### Stateful management of chat history[​](#stateful-management-of-chat-history "Direct link to Stateful management of chat history")

Here we've gone over how to add application logic for incorporating historical outputs, but we're still manually updating the chat history and inserting it into each input. In a real Q&A application we'll want some way of persisting chat history and some way of automatically inserting and updating it.

For this we can use:

*   [BaseChatMessageHistory](https://api.python.langchain.com/en/latest/langchain_api_reference.html#module-langchain.memory): Store chat history.
*   [RunnableWithMessageHistory](/v0.2/docs/how_to/message_history/): Wrapper for an LCEL chain and a `BaseChatMessageHistory` that handles injecting chat history into inputs and updating it after each invocation.

For a detailed walkthrough of how to use these classes together to create a stateful conversational chain, head to the [How to add message history (memory)](/v0.2/docs/how_to/message_history/) LCEL page.

Below, we implement a simple example of the second option, in which chat histories are stored in a simple dict. LangChain manages memory integrations with [Redis](/v0.2/docs/integrations/memory/redis_chat_message_history/) and other technologies to provide for more robust persistence.

Instances of `RunnableWithMessageHistory` manage the chat history for you. They accept a config with a key (`"session_id"` by default) that specifies what conversation history to fetch and prepend to the input, and append the output to the same conversation history. Below is an example:

    from langchain_community.chat_message_histories import ChatMessageHistoryfrom langchain_core.chat_history import BaseChatMessageHistoryfrom langchain_core.runnables.history import RunnableWithMessageHistorystore = {}def get_session_history(session_id: str) -> BaseChatMessageHistory:    if session_id not in store:        store[session_id] = ChatMessageHistory()    return store[session_id]conversational_rag_chain = RunnableWithMessageHistory(    rag_chain,    get_session_history,    input_messages_key="input",    history_messages_key="chat_history",    output_messages_key="answer",)

**API Reference:**[ChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.ChatMessageHistory.html) | [BaseChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.BaseChatMessageHistory.html) | [RunnableWithMessageHistory](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.history.RunnableWithMessageHistory.html)

    conversational_rag_chain.invoke(    {"input": "What is Task Decomposition?"},    config={        "configurable": {"session_id": "abc123"}    },  # constructs a key "abc123" in `store`.)["answer"]

    'Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable. Techniques like Chain of Thought (CoT) and Tree of Thoughts help models decompose hard tasks into multiple manageable subtasks. This process allows agents to plan ahead and tackle intricate tasks effectively.'

    conversational_rag_chain.invoke(    {"input": "What are common ways of doing it?"},    config={"configurable": {"session_id": "abc123"}},)["answer"]

    'Task decomposition can be achieved through various methods such as using Language Model (LLM) with simple prompting, task-specific instructions tailored to the specific task at hand, or incorporating human inputs to break down the task into smaller components. These approaches help in guiding agents to think step by step and decompose complex tasks into more manageable subgoals.'

The conversation history can be inspected in the `store` dict:

    for message in store["abc123"].messages:    if isinstance(message, AIMessage):        prefix = "AI"    else:        prefix = "User"    print(f"{prefix}: {message.content}\n")

    User: What is Task Decomposition?AI: Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable. Techniques like Chain of Thought (CoT) and Tree of Thoughts help models decompose hard tasks into multiple manageable subtasks. This process allows agents to plan ahead and tackle intricate tasks effectively.User: What are common ways of doing it?AI: Task decomposition can be achieved through various methods such as using Language Model (LLM) with simple prompting, task-specific instructions tailored to the specific task at hand, or incorporating human inputs to break down the task into smaller components. These approaches help in guiding agents to think step by step and decompose complex tasks into more manageable subgoals.

### Tying it together[​](#tying-it-together "Direct link to Tying it together")

![](/v0.2/assets/images/conversational_retrieval_chain-5c7a96abe29e582bc575a0a0d63f86b0.png)

For convenience, we tie together all of the necessary steps in a single code cell:

    import bs4from langchain.chains import create_history_aware_retriever, create_retrieval_chainfrom langchain.chains.combine_documents import create_stuff_documents_chainfrom langchain_chroma import Chromafrom langchain_community.chat_message_histories import ChatMessageHistoryfrom langchain_community.document_loaders import WebBaseLoaderfrom langchain_core.chat_history import BaseChatMessageHistoryfrom langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholderfrom langchain_core.runnables.history import RunnableWithMessageHistoryfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain_text_splitters import RecursiveCharacterTextSplitterllm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)### Construct retriever ###loader = WebBaseLoader(    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),    bs_kwargs=dict(        parse_only=bs4.SoupStrainer(            class_=("post-content", "post-title", "post-header")        )    ),)docs = loader.load()text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)splits = text_splitter.split_documents(docs)vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())retriever = vectorstore.as_retriever()### Contextualize question ###contextualize_q_system_prompt = (    "Given a chat history and the latest user question "    "which might reference context in the chat history, "    "formulate a standalone question which can be understood "    "without the chat history. Do NOT answer the question, "    "just reformulate it if needed and otherwise return it as is.")contextualize_q_prompt = ChatPromptTemplate.from_messages(    [        ("system", contextualize_q_system_prompt),        MessagesPlaceholder("chat_history"),        ("human", "{input}"),    ])history_aware_retriever = create_history_aware_retriever(    llm, retriever, contextualize_q_prompt)### Answer question ###system_prompt = (    "You are an assistant for question-answering tasks. "    "Use the following pieces of retrieved context to answer "    "the question. If you don't know the answer, say that you "    "don't know. Use three sentences maximum and keep the "    "answer concise."    "\n\n"    "{context}")qa_prompt = ChatPromptTemplate.from_messages(    [        ("system", system_prompt),        MessagesPlaceholder("chat_history"),        ("human", "{input}"),    ])question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)### Statefully manage chat history ###store = {}def get_session_history(session_id: str) -> BaseChatMessageHistory:    if session_id not in store:        store[session_id] = ChatMessageHistory()    return store[session_id]conversational_rag_chain = RunnableWithMessageHistory(    rag_chain,    get_session_history,    input_messages_key="input",    history_messages_key="chat_history",    output_messages_key="answer",)

**API Reference:**[create\_history\_aware\_retriever](https://api.python.langchain.com/en/latest/chains/langchain.chains.history_aware_retriever.create_history_aware_retriever.html) | [create\_retrieval\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.retrieval.create_retrieval_chain.html) | [create\_stuff\_documents\_chain](https://api.python.langchain.com/en/latest/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html) | [ChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.ChatMessageHistory.html) | [WebBaseLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) | [BaseChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.BaseChatMessageHistory.html) | [ChatPromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.ChatPromptTemplate.html) | [MessagesPlaceholder](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.MessagesPlaceholder.html) | [RunnableWithMessageHistory](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.history.RunnableWithMessageHistory.html) | [ChatOpenAI](https://api.python.langchain.com/en/latest/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html) | [OpenAIEmbeddings](https://api.python.langchain.com/en/latest/embeddings/langchain_openai.embeddings.base.OpenAIEmbeddings.html) | [RecursiveCharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html)

    conversational_rag_chain.invoke(    {"input": "What is Task Decomposition?"},    config={        "configurable": {"session_id": "abc123"}    },  # constructs a key "abc123" in `store`.)["answer"]

    'Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. It involves transforming big tasks into multiple manageable tasks to facilitate problem-solving. Different methods like Chain of Thought and Tree of Thoughts can be employed to decompose tasks effectively.'

    conversational_rag_chain.invoke(    {"input": "What are common ways of doing it?"},    config={"configurable": {"session_id": "abc123"}},)["answer"]

    'Task decomposition can be achieved through various methods such as using prompting techniques like "Steps for XYZ" or "What are the subgoals for achieving XYZ?", providing task-specific instructions like "Write a story outline," or incorporating human inputs to break down complex tasks into smaller components. These approaches help in organizing thoughts and planning ahead for successful task completion.'

Agents[​](#agents "Direct link to Agents")
------------------------------------------

Agents leverage the reasoning capabilities of LLMs to make decisions during execution. Using agents allow you to offload some discretion over the retrieval process. Although their behavior is less predictable than chains, they offer some advantages in this context:

*   Agents generate the input to the retriever directly, without necessarily needing us to explicitly build in contextualization, as we did above;
*   Agents can execute multiple retrieval steps in service of a query, or refrain from executing a retrieval step altogether (e.g., in response to a generic greeting from a user).

### Retrieval tool[​](#retrieval-tool "Direct link to Retrieval tool")

Agents can access "tools" and manage their execution. In this case, we will convert our retriever into a LangChain tool to be wielded by the agent:

    from langchain.tools.retriever import create_retriever_tooltool = create_retriever_tool(    retriever,    "blog_post_retriever",    "Searches and returns excerpts from the Autonomous Agents blog post.",)tools = [tool]

**API Reference:**[create\_retriever\_tool](https://api.python.langchain.com/en/latest/tools/langchain_core.tools.create_retriever_tool.html)

Tools are LangChain [Runnables](/v0.2/docs/concepts/#langchain-expression-language-lcel), and implement the usual interface:

    tool.invoke("task decomposition")

    'Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n\nFig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nFig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'

### Agent constructor[​](#agent-constructor "Direct link to Agent constructor")

Now that we have defined the tools and the LLM, we can create the agent. We will be using [LangGraph](/v0.2/docs/concepts/#langgraph) to construct the agent. Currently we are using a high level interface to construct the agent, but the nice thing about LangGraph is that this high-level interface is backed by a low-level, highly controllable API in case you want to modify the agent logic.

    from langgraph.prebuilt import create_react_agentagent_executor = create_react_agent(llm, tools)

We can now try it out. Note that so far it is not stateful (we still need to add in memory)

    query = "What is Task Decomposition?"for s in agent_executor.stream(    {"messages": [HumanMessage(content=query)]},):    print(s)    print("----")

    Error in LangChainTracer.on_tool_end callback: TracerException("Found chain run at ID 1a50f4da-34a7-44af-8cbb-c67c90c9619e, but expected {'tool'} run.")``````output{'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_1ZkTWsLYIlKZ1uMyIQGUuyJx', 'function': {'arguments': '{"query":"Task Decomposition"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 19, 'prompt_tokens': 68, 'total_tokens': 87}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-dddbe2d2-2355-4ca5-9961-1ceb39d78cf9-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'Task Decomposition'}, 'id': 'call_1ZkTWsLYIlKZ1uMyIQGUuyJx'}])]}}----{'tools': {'messages': [ToolMessage(content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nFig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', name='blog_post_retriever', tool_call_id='call_1ZkTWsLYIlKZ1uMyIQGUuyJx')]}}----{'agent': {'messages': [AIMessage(content='Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. This approach helps in managing and solving difficult tasks by dividing them into more manageable components. One common method of task decomposition is the Chain of Thought (CoT) technique, where models are instructed to think step by step to decompose hard tasks into smaller steps. Another extension of CoT is the Tree of Thoughts, which explores multiple reasoning possibilities at each step and generates multiple thoughts per step, creating a tree structure. Task decomposition can be facilitated by using simple prompts, task-specific instructions, or human inputs.', response_metadata={'token_usage': {'completion_tokens': 119, 'prompt_tokens': 636, 'total_tokens': 755}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-4a701854-97f2-4ec2-b6e1-73410911fa72-0')]}}----

LangGraph comes with built in persistence, so we don't need to use ChatMessageHistory! Rather, we can pass in a checkpointer to our LangGraph agent directly

    from langgraph.checkpoint.sqlite import SqliteSavermemory = SqliteSaver.from_conn_string(":memory:")agent_executor = create_react_agent(llm, tools, checkpointer=memory)

This is all we need to construct a conversational RAG agent.

Let's observe its behavior. Note that if we input a query that does not require a retrieval step, the agent does not execute one:

    config = {"configurable": {"thread_id": "abc123"}}for s in agent_executor.stream(    {"messages": [HumanMessage(content="Hi! I'm bob")]}, config=config):    print(s)    print("----")

    {'agent': {'messages': [AIMessage(content='Hello Bob! How can I assist you today?', response_metadata={'token_usage': {'completion_tokens': 11, 'prompt_tokens': 67, 'total_tokens': 78}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-022806f0-eb26-4c87-9132-ed2fcc6c21ea-0')]}}----

Further, if we input a query that does require a retrieval step, the agent generates the input to the tool:

    query = "What is Task Decomposition?"for s in agent_executor.stream(    {"messages": [HumanMessage(content=query)]}, config=config):    print(s)    print("----")

    {'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_DdAAJJgGIQOZQgKVE4duDyML', 'function': {'arguments': '{"query":"Task Decomposition"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 19, 'prompt_tokens': 91, 'total_tokens': 110}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-acc3c903-4f6f-48dd-8b36-f6f3b80d0856-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'Task Decomposition'}, 'id': 'call_DdAAJJgGIQOZQgKVE4duDyML'}])]}}----``````outputError in LangChainTracer.on_tool_end callback: TracerException("Found chain run at ID 9a7ba580-ec91-412d-9649-1b5cbf5ae7bc, but expected {'tool'} run.")``````output{'tools': {'messages': [ToolMessage(content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nFig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.', name='blog_post_retriever', tool_call_id='call_DdAAJJgGIQOZQgKVE4duDyML')]}}----

Above, instead of inserting our query verbatim into the tool, the agent stripped unnecessary words like "what" and "is".

This same principle allows the agent to use the context of the conversation when necessary:

    query = "What according to the blog post are common ways of doing it? redo the search"for s in agent_executor.stream(    {"messages": [HumanMessage(content=query)]}, config=config):    print(s)    print("----")

    {'agent': {'messages': [AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_KvoiamnLfGEzMeEMlV3u0TJ7', 'function': {'arguments': '{"query":"common ways of task decomposition"}', 'name': 'blog_post_retriever'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 21, 'prompt_tokens': 930, 'total_tokens': 951}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-dd842071-6dbd-4b68-8657-892eaca58638-0', tool_calls=[{'name': 'blog_post_retriever', 'args': {'query': 'common ways of task decomposition'}, 'id': 'call_KvoiamnLfGEzMeEMlV3u0TJ7'}])]}}----{'action': {'messages': [ToolMessage(content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n\nFig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nResources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\n\n(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.', name='blog_post_retriever', id='c749bb8e-c8e0-4fa3-bc11-3e2e0651880b', tool_call_id='call_KvoiamnLfGEzMeEMlV3u0TJ7')]}}----{'agent': {'messages': [AIMessage(content='According to the blog post, common ways of task decomposition include:\n\n1. Using language models with simple prompting like "Steps for XYZ" or "What are the subgoals for achieving XYZ?"\n2. Utilizing task-specific instructions, for example, using "Write a story outline" for writing a novel.\n3. Involving human inputs in the task decomposition process.\n\nThese methods help in breaking down complex tasks into smaller and more manageable steps, facilitating better planning and execution of the overall task.', response_metadata={'token_usage': {'completion_tokens': 100, 'prompt_tokens': 1475, 'total_tokens': 1575}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_3b956da36b', 'finish_reason': 'stop', 'logprobs': None}, id='run-98b765b3-f1a6-4c9a-ad0f-2db7950b900f-0')]}}----

Note that the agent was able to infer that "it" in our query refers to "task decomposition", and generated a reasonable search query as a result-- in this case, "common ways of task decomposition".

### Tying it together[​](#tying-it-together-1 "Direct link to Tying it together")

For convenience, we tie together all of the necessary steps in a single code cell:

    import bs4from langchain.agents import AgentExecutor, create_tool_calling_agentfrom langchain.tools.retriever import create_retriever_toolfrom langchain_chroma import Chromafrom langchain_community.chat_message_histories import ChatMessageHistoryfrom langchain_community.document_loaders import WebBaseLoaderfrom langchain_core.chat_history import BaseChatMessageHistoryfrom langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholderfrom langchain_core.runnables.history import RunnableWithMessageHistoryfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom langgraph.checkpoint.sqlite import SqliteSaverfrom langgraph.prebuilt import create_react_agentmemory = SqliteSaver.from_conn_string(":memory:")llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)### Construct retriever ###loader = WebBaseLoader(    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),    bs_kwargs=dict(        parse_only=bs4.SoupStrainer(            class_=("post-content", "post-title", "post-header")        )    ),)docs = loader.load()text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)splits = text_splitter.split_documents(docs)vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())retriever = vectorstore.as_retriever()### Build retriever tool ###tool = create_retriever_tool(    retriever,    "blog_post_retriever",    "Searches and returns excerpts from the Autonomous Agents blog post.",)tools = [tool]agent_executor = create_react_agent(llm, tools, checkpointer=memory)

**API Reference:**[AgentExecutor](https://api.python.langchain.com/en/latest/agents/langchain.agents.agent.AgentExecutor.html) | [create\_tool\_calling\_agent](https://api.python.langchain.com/en/latest/agents/langchain.agents.tool_calling_agent.base.create_tool_calling_agent.html) | [create\_retriever\_tool](https://api.python.langchain.com/en/latest/tools/langchain_core.tools.create_retriever_tool.html) | [ChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.ChatMessageHistory.html) | [WebBaseLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) | [BaseChatMessageHistory](https://api.python.langchain.com/en/latest/chat_history/langchain_core.chat_history.BaseChatMessageHistory.html) | [ChatPromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.ChatPromptTemplate.html) | [MessagesPlaceholder](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.MessagesPlaceholder.html) | [RunnableWithMessageHistory](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.history.RunnableWithMessageHistory.html) | [ChatOpenAI](https://api.python.langchain.com/en/latest/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html) | [OpenAIEmbeddings](https://api.python.langchain.com/en/latest/embeddings/langchain_openai.embeddings.base.OpenAIEmbeddings.html) | [RecursiveCharacterTextSplitter](https://api.python.langchain.com/en/latest/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html)

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

We've covered the steps to build a basic conversational Q&A application:

*   We used chains to build a predictable application that generates search queries for each user input;
*   We used agents to build an application that "decides" when and how to generate search queries.

To explore different types of retrievers and retrieval strategies, visit the [retrievers](/v0.2/docs/how_to/#retrievers) section of the how-to guides.

For a detailed walkthrough of LangChain's conversation memory abstractions, visit the [How to add message history (memory)](/v0.2/docs/how_to/message_history/) LCEL page.

To learn more about agents, head to the [Agents Modules](/v0.2/docs/tutorials/agents/).

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/tutorials/qa_chat_history.ipynb)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

Build a Chatbot

](/v0.2/docs/tutorials/chatbot/)[

Next

Build an Extraction Chain

](/v0.2/docs/tutorials/extraction/)

*   [Setup](#setup)
    *   [Dependencies](#dependencies)
    *   [LangSmith](#langsmith)
*   [Chains](#chains)
    *   [Adding chat history](#adding-chat-history)
    *   [Tying it together](#tying-it-together)
*   [Agents](#agents)
    *   [Retrieval tool](#retrieval-tool)
    *   [Agent constructor](#agent-constructor)
    *   [Tying it together](#tying-it-together-1)
*   [Next steps](#next-steps)