!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

You are viewing the **preview** v0.2 docs. View the **stable** v0.1 docs [here](/v0.1/docs/get_started/introduction/). Leave feedback on the v0.2 docs [here](https://github.com/langchain-ai/langchainjs/discussions/5386).

[

![🦜️🔗 Langchain](/v0.2/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.2/img/brand/wordmark-dark.png)

](/v0.2/)[Integrations](/v0.2/docs/integrations/platforms/)[API Reference](https://v02.api.js.langchain.com)

[More](#)

*   [People](/v0.2/docs/people/)
*   [Community](/v0.2/docs/community)
*   [Tutorials](/v0.2/docs/additional_resources/tutorials)
*   [Contributing](/v0.2/docs/contributing)

[v0.2](#)

*   [v0.2](/v0.2/docs/introduction)
*   [v0.1](https://js.langchain.com/v0.1/docs/get_started/introduction)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Introduction](/v0.2/docs/introduction)
*   [Tutorials](/v0.2/docs/tutorials/)
    
    *   [Build a Question Answering application over a Graph Database](/v0.2/docs/tutorials/graph)
    *   [Tutorials](/v0.2/docs/tutorials/)
    *   [Build a Simple LLM Application](/v0.2/docs/tutorials/llm_chain)
    *   [Build a Query Analysis System](/v0.2/docs/tutorials/query_analysis)
    *   [Build a Chatbot](/v0.2/docs/tutorials/chatbot)
    *   [Build an Agent](/v0.2/docs/tutorials/agents)
    *   [Build an Extraction Chain](/v0.2/docs/tutorials/extraction)
    *   [Summarize Text](/v0.2/docs/tutorials/summarization)
    *   [Tagging](/v0.2/docs/tutorials/classification)
    *   [Build a Local RAG Application](/v0.2/docs/tutorials/local_rag)
    *   [Conversational RAG](/v0.2/docs/tutorials/qa_chat_history)
    *   [Build a Retrieval Augmented Generation (RAG) App](/v0.2/docs/tutorials/rag)
    *   [Build a Question/Answering system over SQL data](/v0.2/docs/tutorials/sql_qa)
*   [How-to guides](/v0.2/docs/how_to/)
    
    *   [How-to guides](/v0.2/docs/how_to/)
    *   [How to use example selectors](/v0.2/docs/how_to/example_selectors)
    *   [Installation](/v0.2/docs/how_to/installation)
    *   [How to stream responses from an LLM](/v0.2/docs/how_to/streaming_llm)
    *   [How to stream chat model responses](/v0.2/docs/how_to/chat_streaming)
    *   [How to embed text data](/v0.2/docs/how_to/embed_text)
    *   [How to use few shot examples in chat models](/v0.2/docs/how_to/few_shot_examples_chat)
    *   [How to cache model responses](/v0.2/docs/how_to/llm_caching)
    *   [How to cache chat model responses](/v0.2/docs/how_to/chat_model_caching)
    *   [How to create a custom LLM class](/v0.2/docs/how_to/custom_llm)
    *   [How to use few shot examples](/v0.2/docs/how_to/few_shot_examples)
    *   [How to use output parsers to parse an LLM response into structured format](/v0.2/docs/how_to/output_parser_structured)
    *   [How to return structured data from a model](/v0.2/docs/how_to/structured_output)
    *   [How to add ad-hoc tool calling capability to LLMs and Chat Models](/v0.2/docs/how_to/tools_prompting)
    *   [How to create a custom chat model class](/v0.2/docs/how_to/custom_chat)
    *   [How to do per-user retrieval](/v0.2/docs/how_to/qa_per_user)
    *   [How to track token usage](/v0.2/docs/how_to/chat_token_usage_tracking)
    *   [How to track token usage](/v0.2/docs/how_to/llm_token_usage_tracking)
    *   [How to pass through arguments from one step to the next](/v0.2/docs/how_to/passthrough)
    *   [How to compose prompts together](/v0.2/docs/how_to/prompts_composition)
    *   [How to use legacy LangChain Agents (AgentExecutor)](/v0.2/docs/how_to/agent_executor)
    *   [How to add values to a chain's state](/v0.2/docs/how_to/assign)
    *   [How to attach runtime arguments to a Runnable](/v0.2/docs/how_to/binding)
    *   [How to cache embedding results](/v0.2/docs/how_to/caching_embeddings)
    *   [How to split by character](/v0.2/docs/how_to/character_text_splitter)
    *   [How to manage memory](/v0.2/docs/how_to/chatbots_memory)
    *   [How to do retrieval](/v0.2/docs/how_to/chatbots_retrieval)
    *   [How to use tools](/v0.2/docs/how_to/chatbots_tools)
    *   [How to split code](/v0.2/docs/how_to/code_splitter)
    *   [How to do retrieval with contextual compression](/v0.2/docs/how_to/contextual_compression)
    *   [How to write a custom retriever class](/v0.2/docs/how_to/custom_retriever)
    *   [How to create custom Tools](/v0.2/docs/how_to/custom_tools)
    *   [How to debug your LLM apps](/v0.2/docs/how_to/debugging)
    *   [How to load CSV data](/v0.2/docs/how_to/document_loader_csv)
    *   [How to write a custom document loader](/v0.2/docs/how_to/document_loader_custom)
    *   [How to load data from a directory](/v0.2/docs/how_to/document_loader_directory)
    *   [How to load PDF files](/v0.2/docs/how_to/document_loader_pdf)
    *   [How to load JSON data](/v0.2/docs/how_to/document_loaders_json)
    *   [How to select examples by length](/v0.2/docs/how_to/example_selectors_length_based)
    *   [How to select examples by similarity](/v0.2/docs/how_to/example_selectors_similarity)
    *   [How to use reference examples](/v0.2/docs/how_to/extraction_examples)
    *   [How to handle long text](/v0.2/docs/how_to/extraction_long_text)
    *   [How to do extraction without using function calling](/v0.2/docs/how_to/extraction_parse)
    *   [Fallbacks](/v0.2/docs/how_to/fallbacks)
    *   [Few Shot Prompt Templates](/v0.2/docs/how_to/few_shot)
    *   [How to run custom functions](/v0.2/docs/how_to/functions)
    *   [How to construct knowledge graphs](/v0.2/docs/how_to/graph_constructing)
    *   [How to map values to a database](/v0.2/docs/how_to/graph_mapping)
    *   [How to improve results with prompting](/v0.2/docs/how_to/graph_prompting)
    *   [How to add a semantic layer over the database](/v0.2/docs/how_to/graph_semantic)
    *   [How to reindex data to keep your vectorstore in-sync with the underlying data source](/v0.2/docs/how_to/indexing)
    *   [How to get log probabilities](/v0.2/docs/how_to/logprobs)
    *   [How to add message history](/v0.2/docs/how_to/message_history)
    *   [How to generate multiple embeddings per document](/v0.2/docs/how_to/multi_vector)
    *   [How to generate multiple queries to retrieve data for](/v0.2/docs/how_to/multiple_queries)
    *   [How to parse JSON output](/v0.2/docs/how_to/output_parser_json)
    *   [How to retry when output parsing errors occur](/v0.2/docs/how_to/output_parser_retry)
    *   [How to parse XML output](/v0.2/docs/how_to/output_parser_xml)
    *   [How to invoke runnables in parallel](/v0.2/docs/how_to/parallel)
    *   [How to retrieve the whole document for a chunk](/v0.2/docs/how_to/parent_document_retriever)
    *   [How to partially format prompt templates](/v0.2/docs/how_to/prompts_partial)
    *   [How to add chat history to a question-answering chain](/v0.2/docs/how_to/qa_chat_history_how_to)
    *   [How to return citations](/v0.2/docs/how_to/qa_citations)
    *   [How to return sources](/v0.2/docs/how_to/qa_sources)
    *   [How to stream from a question-answering chain](/v0.2/docs/how_to/qa_streaming)
    *   [How to construct filters](/v0.2/docs/how_to/query_constructing_filters)
    *   [How to add examples to the prompt](/v0.2/docs/how_to/query_few_shot)
    *   [How to deal with high cardinality categorical variables](/v0.2/docs/how_to/query_high_cardinality)
    *   [How to handle multiple queries](/v0.2/docs/how_to/query_multiple_queries)
    *   [How to handle multiple retrievers](/v0.2/docs/how_to/query_multiple_retrievers)
    *   [How to handle cases where no queries are generated](/v0.2/docs/how_to/query_no_queries)
    *   [How to recursively split text by characters](/v0.2/docs/how_to/recursive_text_splitter)
    *   [How to reduce retrieval latency](/v0.2/docs/how_to/reduce_retrieval_latency)
    *   [How to route execution within a chain](/v0.2/docs/how_to/routing)
    *   [How to chain runnables](/v0.2/docs/how_to/sequence)
    *   [How to split text by tokens](/v0.2/docs/how_to/split_by_token)
    *   [How to deal with large databases](/v0.2/docs/how_to/sql_large_db)
    *   [How to use prompting to improve results](/v0.2/docs/how_to/sql_prompting)
    *   [How to do query validation](/v0.2/docs/how_to/sql_query_checking)
    *   [How to stream](/v0.2/docs/how_to/streaming)
    *   [How to create a time-weighted retriever](/v0.2/docs/how_to/time_weighted_vectorstore)
    *   [How to use a chat model to call tools](/v0.2/docs/how_to/tool_calling)
    *   [How to call tools with multi-modal data](/v0.2/docs/how_to/tool_calls_multi_modal)
    *   [How to use LangChain tools](/v0.2/docs/how_to/tools_builtin)
    *   [How use a vector store to retrieve data](/v0.2/docs/how_to/vectorstore_retriever)
    *   [How to create and query vector stores](/v0.2/docs/how_to/vectorstores)
*   [Conceptual guide](/v0.2/docs/concepts)
*   Ecosystem
    
    *   [🦜🛠️ LangSmith](/v0.2/docs/langsmith/)
        
    *   [🦜🕸️LangGraph.js](/v0.2/docs/langgraph)
*   Versions
    
    *   [Overview](/v0.2/docs/versions/overview)
    *   [v0.2](/v0.2/docs/versions/v0_2)
    *   [Release Policy](/v0.2/docs/versions/release_policy)
    *   [Packages](/v0.2/docs/versions/packages)
*   [Security](/v0.2/docs/security)

*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to use reference examples

On this page

How to use reference examples
=============================

Prerequisites

This guide assumes familiarity with the following:

*   [Extraction](/v0.2/docs/tutorials/extraction)

The quality of extraction can often be improved by providing reference examples to the LLM.

tip

While this tutorial focuses how to use examples with a tool calling model, this technique is generally applicable, and will work also with JSON more or prompt based techniques.

We’ll use OpenAI’s GPT-4 this time for their robust support for `ToolMessages`:

tip

See [this section for general instructions on installing integration packages](/v0.2/docs/how_to/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/openai zod uuid

    yarn add @langchain/openai zod uuid

    pnpm add @langchain/openai zod uuid

Let’s define a prompt:

    import {  ChatPromptTemplate,  MessagesPlaceholder,} from "@langchain/core/prompts";const SYSTEM_PROMPT_TEMPLATE = `You are an expert extraction algorithm.Only extract relevant information from the text.If you do not know the value of an attribute asked to extract, you may omit the attribute's value.`;// Define a custom prompt to provide instructions and any additional context.// 1) You can add examples into the prompt template to improve extraction quality// 2) Introduce additional parameters to take context into account (e.g., include metadata//    about the document from which the text was extracted.)const prompt = ChatPromptTemplate.fromMessages([  ["system", SYSTEM_PROMPT_TEMPLATE],  // ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓  new MessagesPlaceholder("examples"),  // ↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑  ["human", "{text}"],]);

Test out the template:

    import { HumanMessage } from "@langchain/core/messages";const promptValue = await prompt.invoke({  text: "this is some text",  examples: [new HumanMessage("testing 1 2 3")],});promptValue.toChatMessages();

    [  SystemMessage {    lc_serializable: true,    lc_kwargs: {      content: "You are an expert extraction algorithm.\n" +        "Only extract relevant information from the text.\n" +        "If you do n"... 87 more characters,      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "You are an expert extraction algorithm.\n" +      "Only extract relevant information from the text.\n" +      "If you do n"... 87 more characters,    name: undefined,    additional_kwargs: {}  },  HumanMessage {    lc_serializable: true,    lc_kwargs: { content: "testing 1 2 3", additional_kwargs: {} },    lc_namespace: [ "langchain_core", "messages" ],    content: "testing 1 2 3",    name: undefined,    additional_kwargs: {}  },  HumanMessage {    lc_serializable: true,    lc_kwargs: { content: "this is some text", additional_kwargs: {} },    lc_namespace: [ "langchain_core", "messages" ],    content: "this is some text",    name: undefined,    additional_kwargs: {}  }]

Define the schema[​](#define-the-schema "Direct link to Define the schema")
---------------------------------------------------------------------------

Let’s re-use the people schema from the quickstart.

    import { z } from "zod";const personSchema = z  .object({    name: z.optional(z.string()).describe("The name of the person"),    hair_color: z      .optional(z.string())      .describe("The color of the person's hair, if known"),    height_in_meters: z      .optional(z.string())      .describe("Height measured in meters"),  })  .describe("Information about a person.");const peopleSchema = z.object({  people: z.array(personSchema),});

Define reference examples[​](#define-reference-examples "Direct link to Define reference examples")
---------------------------------------------------------------------------------------------------

Examples can be defined as a list of input-output pairs.

Each example contains an example `input` text and an example `output` showing what should be extracted from the text.

info

The below example is a bit more advanced - the format of the example needs to match the API used (e.g., tool calling or JSON mode etc.).

Here, the formatted examples will match the format expected for the OpenAI tool calling API since that’s what we’re using.

To provide reference examples to the model, we will mock out a fake chat history containing successful usages of the given tool. Because the model can choose to call multiple tools at once (or the same tool multiple times), the example’s outputs are an array:

    import {  AIMessage,  type BaseMessage,  HumanMessage,  ToolMessage,} from "@langchain/core/messages";import { v4 as uuid } from "uuid";type OpenAIToolCall = {  id: string;  type: "function";  function: {    name: string;    arguments: string;  };};type Example = {  input: string;  toolCallOutputs: Record<string, any>[];};/** * This function converts an example into a list of messages that can be fed into an LLM. * * This code serves as an adapter that transforms our example into a list of messages * that can be processed by a chat model. * * The list of messages for each example includes: * * 1) HumanMessage: This contains the content from which information should be extracted. * 2) AIMessage: This contains the information extracted by the model. * 3) ToolMessage: This provides confirmation to the model that the tool was requested correctly. * * The inclusion of ToolMessage is necessary because some chat models are highly optimized for agents, * making them less suitable for an extraction use case. */function toolExampleToMessages(example: Example): BaseMessage[] {  const openAIToolCalls: OpenAIToolCall[] = example.toolCallOutputs.map(    (output) => {      return {        id: uuid(),        type: "function",        function: {          // The name of the function right now corresponds          // to the passed name.          name: "extract",          arguments: JSON.stringify(output),        },      };    }  );  const messages: BaseMessage[] = [    new HumanMessage(example.input),    new AIMessage({      content: "",      additional_kwargs: { tool_calls: openAIToolCalls },    }),  ];  const toolMessages = openAIToolCalls.map((toolCall, i) => {    // Return the mocked successful result for a given tool call.    return new ToolMessage({      content: "You have correctly called this tool.",      tool_call_id: toolCall.id,    });  });  return messages.concat(toolMessages);}

Next let’s define our examples and then convert them into message format.

    const examples: Example[] = [  {    input:      "The ocean is vast and blue. It's more than 20,000 feet deep. There are many fish in it.",    toolCallOutputs: [{}],  },  {    input: "Fiona traveled far from France to Spain.",    toolCallOutputs: [      {        name: "Fiona",      },    ],  },];const exampleMessages = [];for (const example of examples) {  exampleMessages.push(...toolExampleToMessages(example));}

    6

Let’s test out the prompt

    const promptValue = await prompt.invoke({  text: "this is some text",  examples: exampleMessages,});promptValue.toChatMessages();

    [  SystemMessage {    lc_serializable: true,    lc_kwargs: {      content: "You are an expert extraction algorithm.\n" +        "Only extract relevant information from the text.\n" +        "If you do n"... 87 more characters,      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "You are an expert extraction algorithm.\n" +      "Only extract relevant information from the text.\n" +      "If you do n"... 87 more characters,    name: undefined,    additional_kwargs: {}  },  HumanMessage {    lc_serializable: true,    lc_kwargs: {      content: "The ocean is vast and blue. It's more than 20,000 feet deep. There are many fish in it.",      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "The ocean is vast and blue. It's more than 20,000 feet deep. There are many fish in it.",    name: undefined,    additional_kwargs: {}  },  AIMessage {    lc_serializable: true,    lc_kwargs: { content: "", additional_kwargs: { tool_calls: [ [Object] ] } },    lc_namespace: [ "langchain_core", "messages" ],    content: "",    name: undefined,    additional_kwargs: {      tool_calls: [        {          id: "8fa4d00d-801f-470e-8737-51ee9dc82259",          type: "function",          function: [Object]        }      ]    }  },  ToolMessage {    lc_serializable: true,    lc_kwargs: {      content: "You have correctly called this tool.",      tool_call_id: "8fa4d00d-801f-470e-8737-51ee9dc82259",      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "You have correctly called this tool.",    name: undefined,    additional_kwargs: {},    tool_call_id: "8fa4d00d-801f-470e-8737-51ee9dc82259"  },  HumanMessage {    lc_serializable: true,    lc_kwargs: {      content: "Fiona traveled far from France to Spain.",      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "Fiona traveled far from France to Spain.",    name: undefined,    additional_kwargs: {}  },  AIMessage {    lc_serializable: true,    lc_kwargs: { content: "", additional_kwargs: { tool_calls: [ [Object] ] } },    lc_namespace: [ "langchain_core", "messages" ],    content: "",    name: undefined,    additional_kwargs: {      tool_calls: [        {          id: "14ad6217-fcbd-47c7-9006-82f612e36c66",          type: "function",          function: [Object]        }      ]    }  },  ToolMessage {    lc_serializable: true,    lc_kwargs: {      content: "You have correctly called this tool.",      tool_call_id: "14ad6217-fcbd-47c7-9006-82f612e36c66",      additional_kwargs: {}    },    lc_namespace: [ "langchain_core", "messages" ],    content: "You have correctly called this tool.",    name: undefined,    additional_kwargs: {},    tool_call_id: "14ad6217-fcbd-47c7-9006-82f612e36c66"  },  HumanMessage {    lc_serializable: true,    lc_kwargs: { content: "this is some text", additional_kwargs: {} },    lc_namespace: [ "langchain_core", "messages" ],    content: "this is some text",    name: undefined,    additional_kwargs: {}  }]

Create an extractor[​](#create-an-extractor "Direct link to Create an extractor")
---------------------------------------------------------------------------------

Here, we’ll create an extractor using **gpt-4**.

    import { ChatOpenAI } from "@langchain/openai";// We will be using tool calling mode, which// requires a tool calling capable model.const llm = new ChatOpenAI({  // Consider benchmarking with the best model you can to get  // a sense of the best possible quality.  model: "gpt-4-0125-preview",  temperature: 0,});// For function/tool calling, we can also supply an name for the schema// to give the LLM additional context about what it's extracting.const extractionRunnable = prompt.pipe(  llm.withStructuredOutput(peopleSchema, { name: "people" }));

Without examples 😿[​](#without-examples "Direct link to Without examples 😿")
------------------------------------------------------------------------------

Notice that even though we’re using `gpt-4`, it’s unreliable with a **very simple** test case!

We run it 5 times below to emphasize this:

    const text = "The solar system is large, but earth has only 1 moon.";for (let i = 0; i < 5; i++) {  const result = await extractionRunnable.invoke({    text,    examples: [],  });  console.log(result);}

    {  people: [ { name: "earth", hair_color: "grey", height_in_meters: "1" } ]}{ people: [ { name: "earth", hair_color: "moon" } ] }{ people: [ { name: "earth", hair_color: "moon" } ] }{ people: [ { name: "earth", hair_color: "1 moon" } ] }{ people: [] }

With examples 😻[​](#with-examples "Direct link to With examples 😻")
---------------------------------------------------------------------

Reference examples help fix the failure!

    const text = "The solar system is large, but earth has only 1 moon.";for (let i = 0; i < 5; i++) {  const result = await extractionRunnable.invoke({    text,    // Example messages from above    examples: exampleMessages,  });  console.log(result);}

    { people: [] }{ people: [] }{ people: [] }{ people: [] }{ people: [] }

    await extractionRunnable.invoke({  text: "My name is Hair-ison. My hair is black. I am 3 meters tall.",  examples: exampleMessages,});

    {  people: [ { name: "Hair-ison", hair_color: "black", height_in_meters: "3" } ]}

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned how to improve extraction quality using few-shot examples.

Next, check out some of the other guides in this section, such as [some tips on how to perform extraction on long text](/v0.2/docs/how_to/extraction_long_text).

* * *

#### Was this page helpful?

  

#### You can leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to select examples by similarity

](/v0.2/docs/how_to/example_selectors_similarity)[

Next

How to handle long text

](/v0.2/docs/how_to/extraction_long_text)

*   [Define the schema](#define-the-schema)
*   [Define reference examples](#define-reference-examples)
*   [Create an extractor](#create-an-extractor)
*   [Without examples 😿](#without-examples)
*   [With examples 😻](#with-examples)
*   [Next steps](#next-steps)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.