!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Use cases](/v0.1/docs/use_cases/)
    
    *   [SQL](/v0.1/docs/use_cases/sql/)
        
    *   [Chatbots](/v0.1/docs/use_cases/chatbots/)
        
    *   [Extraction](/v0.1/docs/use_cases/extraction/)
        
    *   [Query Analysis](/v0.1/docs/use_cases/query_analysis/)
        
        *   [Quickstart](/v0.1/docs/use_cases/query_analysis/quickstart/)
        *   [Techniques](/v0.1/docs/use_cases/query_analysis/techniques/decomposition/)
            
            *   [Decomposition](/v0.1/docs/use_cases/query_analysis/techniques/decomposition/)
            *   [Expansion](/v0.1/docs/use_cases/query_analysis/techniques/expansion/)
            *   [Hypothetical Document Embeddings](/v0.1/docs/use_cases/query_analysis/techniques/hyde/)
            *   [Routing](/v0.1/docs/use_cases/query_analysis/techniques/routing/)
            *   [Step Back Prompting](/v0.1/docs/use_cases/query_analysis/techniques/step_back/)
            *   [Structuring](/v0.1/docs/use_cases/query_analysis/techniques/structuring/)
        *   [How-To Guides](/v0.1/docs/use_cases/query_analysis/how_to/few_shot/)
            
    *   [Q&A with RAG](/v0.1/docs/use_cases/question_answering/)
        
    *   [Tool use](/v0.1/docs/use_cases/tool_use/)
        
    *   [Interacting with APIs](/v0.1/docs/use_cases/api/)
    *   [Tabular Question Answering](/v0.1/docs/use_cases/tabular/)
    *   [Graphs](/v0.1/docs/use_cases/graph/)
        
    *   [Summarization](/v0.1/docs/use_cases/summarization/)
    *   [Agent Simulations](/v0.1/docs/use_cases/agent_simulations/)
        
    *   [Autonomous Agents](/v0.1/docs/use_cases/autonomous_agents/)
        
    *   [Code Understanding](/v0.1/docs/use_cases/code_understanding/)
    *   [Audio/Video Structured Extraction](/v0.1/docs/use_cases/media/)

*   [](/v0.1/)
*   [Use cases](/v0.1/docs/use_cases/)
*   [Query Analysis](/v0.1/docs/use_cases/query_analysis/)
*   Techniques
*   Hypothetical Document Embeddings

On this page

Hypothetical Document Embeddings
================================

If we’re working with a similarity search-based index, like a vector store, then searching on raw questions may not work well because their embeddings may not be very similar to those of the relevant documents. Instead it might help to have the model generate a hypothetical relevant document, and then use that to perform similarity search. This is the key idea behind [Hypothetical Document Embedding, or HyDE](https://arxiv.org/pdf/2212.10496.pdf).

Let’s take a look at how we might perform search via hypothetical documents for our Q&A bot over the LangChain YouTube videos.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

#### Install dependencies[​](#install-dependencies "Direct link to Install dependencies")

tip

See [this section for general instructions on installing integration packages](/v0.1/docs/get_started/installation/#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/core zod

    yarn add @langchain/core zod

    pnpm add @langchain/core zod

#### Set environment variables[​](#set-environment-variables "Direct link to Set environment variables")

    # Optional, use LangSmith for best-in-class observabilityLANGSMITH_API_KEY=your-api-keyLANGCHAIN_TRACING_V2=true

Hypothetical document generation[​](#hypothetical-document-generation "Direct link to Hypothetical document generation")
------------------------------------------------------------------------------------------------------------------------

Ultimately generating a relevant hypothetical document reduces to trying to answer the user question. Since we’re desiging a Q&A bot for LangChain YouTube videos, we’ll provide some basic context about LangChain and prompt the model to use a more pedantic style so that we get more realistic hypothetical documents:

### Pick your chat model:

*   OpenAI
*   Anthropic
*   FireworksAI
*   MistralAI

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/openai 

    yarn add @langchain/openai 

    pnpm add @langchain/openai 

#### Add environment variables

    OPENAI_API_KEY=your-api-key

#### Instantiate the model

    import { ChatOpenAI } from "@langchain/openai";const llm = new ChatOpenAI({  model: "gpt-3.5-turbo-0125",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/anthropic 

    yarn add @langchain/anthropic 

    pnpm add @langchain/anthropic 

#### Add environment variables

    ANTHROPIC_API_KEY=your-api-key

#### Instantiate the model

    import { ChatAnthropic } from "@langchain/anthropic";const llm = new ChatAnthropic({  model: "claude-3-sonnet-20240229",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/community 

    yarn add @langchain/community 

    pnpm add @langchain/community 

#### Add environment variables

    FIREWORKS_API_KEY=your-api-key

#### Instantiate the model

    import { ChatFireworks } from "@langchain/community/chat_models/fireworks";const llm = new ChatFireworks({  model: "accounts/fireworks/models/firefunction-v1",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/mistralai 

    yarn add @langchain/mistralai 

    pnpm add @langchain/mistralai 

#### Add environment variables

    MISTRAL_API_KEY=your-api-key

#### Instantiate the model

    import { ChatMistralAI } from "@langchain/mistralai";const llm = new ChatMistralAI({  model: "mistral-large-latest",  temperature: 0});

    import { StringOutputParser } from "@langchain/core/output_parsers";import { ChatPromptTemplate } from "@langchain/core/prompts";const system = `You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.LangSmith is a platform that makes it easy to trace and test LLM applications.Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question. `;const prompt = ChatPromptTemplate.fromMessages([  ["system", system],  ["human", "{question}"],]);const qaNoContext = prompt.pipe(llm).pipe(new StringOutputParser());

    const answer = await qaNoContext.invoke({  question:    "how to use multi-modal models in a chain and turn chain into a rest api",});console.log(answer);

    To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabilities of LangChain, LangGraph, and LangServe. Here's a step-by-step guide on how to achieve this:1. **Set up LangChain**: Start by installing LangChain, LangGraph, and LangServe in your Python environment. You can do this using pip:```bashpip install langchain langgraph langserve```2. **Build a Multi-Modal Model**: Create your multi-modal model using LangChain. LangChain provides integrations with various deep learning frameworks like TensorFlow, PyTorch, and Hugging Face Transformers. You can easily compose different modalities (text, image, audio, etc.) in your model.3. **Use LangGraph for Stateful Multi-Actor Applications**: If your multi-modal model requires stateful interactions between different actors, you can use LangGraph to build such applications. LangGraph simplifies the process of managing state and interactions in your LLM application.4. **Deploy as a REST API using LangServe**: Once you have built your multi-modal model and defined the interactions using LangGraph, you can deploy your chain as a REST API using LangServe. LangServe makes it easy to expose your LangChain application as a web service, allowing users to interact with your model through HTTP requests.5. **Define Endpoints**: In your LangServe application, define the endpoints that correspond to different functionalities of your multi-modal model. For example, you can have endpoints for text input, image input, audio input, etc.6. **Handle Requests**: Implement the logic to handle incoming requests in your LangServe application. Parse the input data, pass it through your multi-modal model, and return the results in the desired format.7. **Start the LangServe Server**: Once you have defined your endpoints and request handling logic, start the LangServe server to make your multi-modal model accessible as a REST API. You can specify the host, port, and other configurations when starting the server.By following these steps, you can effectively use multi-modal models in a chain and expose it as a REST API using LangChain, LangGraph, and LangServe. This approach allows you to build complex LLM applications with stateful interactions and make them accessible to users through a web interface.

Returning the hypothetical document and original question[​](#returning-the-hypothetical-document-and-original-question "Direct link to Returning the hypothetical document and original question")
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To increase our recall we may want to retrieve documents based on both the hypothetical document and the original question. We can easily return both like so:

    import { RunnablePassthrough } from "@langchain/core/runnables";const hydeChain = RunnablePassthrough.assign({  hypotheticalDocument: qaNoContext,});await hydeChain.invoke({  question:    "how to use multi-modal models in a chain and turn chain into a rest api",});

    {  question: "how to use multi-modal models in a chain and turn chain into a rest api",  hypotheticalDocument: "To use multi-modal models in a chain and turn the chain into a REST API, you can leverage the capabi"... 1920 more characters}

Using function-calling to get structured output[​](#using-function-calling-to-get-structured-output "Direct link to Using function-calling to get structured output")
---------------------------------------------------------------------------------------------------------------------------------------------------------------------

If we were composing this technique with other query analysis techniques, we’d likely be using function calling to get out structured query objects. We can use function-calling for HyDE like so:

    import { z } from "zod";const querySchema = z.object({  answer: z    .string()    .describe(      "Answer the user question as best you can. Answer as though you were writing a tutorial that addressed the user question."    ),});const system = `You are an expert about a set of software for building LLM-powered applications called LangChain, LangGraph, LangServe, and LangSmith.LangChain is a Python framework that provides a large set of integrations that can easily be composed to build LLM applications.LangGraph is a Python package built on top of LangChain that makes it easy to build stateful, multi-actor LLM applications.LangServe is a Python package built on top of LangChain that makes it easy to deploy a LangChain application as a REST API.LangSmith is a platform that makes it easy to trace and test LLM applications.`;const prompt = ChatPromptTemplate.fromMessages([  ["system", system],  ["human", "{question}"],]);const llmWithTools = llm.withStructuredOutput(querySchema, {  name: "Query",});const hydeChain = prompt.pipe(llmWithTools);await hydeChain.invoke({  question:    "how to use multi-modal models in a chain and turn chain into a rest api",});

    {  answer: "To use multi-modal models in a chain and turn the chain into a REST API, you can follow these steps:"... 713 more characters}

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

Expansion

](/v0.1/docs/use_cases/query_analysis/techniques/expansion/)[

Next

Routing

](/v0.1/docs/use_cases/query_analysis/techniques/routing/)

*   [Setup](#setup)
*   [Hypothetical document generation](#hypothetical-document-generation)
*   [Returning the hypothetical document and original question](#returning-the-hypothetical-document-and-original-question)
*   [Using function-calling to get structured output](#using-function-calling-to-get-structured-output)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.