!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

You are viewing the **preview** v0.2 docs. View the **stable** v0.1 docs [here](/v0.1/docs/get_started/introduction/). Leave feedback on the v0.2 docs [here](https://github.com/langchain-ai/langchainjs/discussions/5386).

[

![🦜️🔗 Langchain](/v0.2/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.2/img/brand/wordmark-dark.png)

](/v0.2/)[Integrations](/v0.2/docs/integrations/platforms/)[API Reference](https://v02.api.js.langchain.com)

[More](#)

*   [People](/v0.2/docs/people/)
*   [Community](/v0.2/docs/community)
*   [Tutorials](/v0.2/docs/additional_resources/tutorials)
*   [Contributing](/v0.2/docs/contributing)

[v0.2](#)

*   [v0.2](/v0.2/docs/introduction)
*   [v0.1](https://js.langchain.com/v0.1/docs/get_started/introduction)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Introduction](/v0.2/docs/introduction)
*   [Tutorials](/v0.2/docs/tutorials/)
    
    *   [Build a Question Answering application over a Graph Database](/v0.2/docs/tutorials/graph)
    *   [Tutorials](/v0.2/docs/tutorials/)
    *   [Build a Simple LLM Application](/v0.2/docs/tutorials/llm_chain)
    *   [Build a Query Analysis System](/v0.2/docs/tutorials/query_analysis)
    *   [Build a Chatbot](/v0.2/docs/tutorials/chatbot)
    *   [Build an Agent](/v0.2/docs/tutorials/agents)
    *   [Build an Extraction Chain](/v0.2/docs/tutorials/extraction)
    *   [Summarize Text](/v0.2/docs/tutorials/summarization)
    *   [Tagging](/v0.2/docs/tutorials/classification)
    *   [Build a Local RAG Application](/v0.2/docs/tutorials/local_rag)
    *   [Conversational RAG](/v0.2/docs/tutorials/qa_chat_history)
    *   [Build a Retrieval Augmented Generation (RAG) App](/v0.2/docs/tutorials/rag)
    *   [Build a Question/Answering system over SQL data](/v0.2/docs/tutorials/sql_qa)
*   [How-to guides](/v0.2/docs/how_to/)
    
    *   [How-to guides](/v0.2/docs/how_to/)
    *   [How to use example selectors](/v0.2/docs/how_to/example_selectors)
    *   [Installation](/v0.2/docs/how_to/installation)
    *   [How to stream responses from an LLM](/v0.2/docs/how_to/streaming_llm)
    *   [How to stream chat model responses](/v0.2/docs/how_to/chat_streaming)
    *   [How to embed text data](/v0.2/docs/how_to/embed_text)
    *   [How to use few shot examples in chat models](/v0.2/docs/how_to/few_shot_examples_chat)
    *   [How to cache model responses](/v0.2/docs/how_to/llm_caching)
    *   [How to cache chat model responses](/v0.2/docs/how_to/chat_model_caching)
    *   [How to create a custom LLM class](/v0.2/docs/how_to/custom_llm)
    *   [How to use few shot examples](/v0.2/docs/how_to/few_shot_examples)
    *   [How to use output parsers to parse an LLM response into structured format](/v0.2/docs/how_to/output_parser_structured)
    *   [How to return structured data from a model](/v0.2/docs/how_to/structured_output)
    *   [How to add ad-hoc tool calling capability to LLMs and Chat Models](/v0.2/docs/how_to/tools_prompting)
    *   [How to create a custom chat model class](/v0.2/docs/how_to/custom_chat)
    *   [How to do per-user retrieval](/v0.2/docs/how_to/qa_per_user)
    *   [How to track token usage](/v0.2/docs/how_to/chat_token_usage_tracking)
    *   [How to track token usage](/v0.2/docs/how_to/llm_token_usage_tracking)
    *   [How to pass through arguments from one step to the next](/v0.2/docs/how_to/passthrough)
    *   [How to compose prompts together](/v0.2/docs/how_to/prompts_composition)
    *   [How to use legacy LangChain Agents (AgentExecutor)](/v0.2/docs/how_to/agent_executor)
    *   [How to add values to a chain's state](/v0.2/docs/how_to/assign)
    *   [How to attach runtime arguments to a Runnable](/v0.2/docs/how_to/binding)
    *   [How to cache embedding results](/v0.2/docs/how_to/caching_embeddings)
    *   [How to split by character](/v0.2/docs/how_to/character_text_splitter)
    *   [How to manage memory](/v0.2/docs/how_to/chatbots_memory)
    *   [How to do retrieval](/v0.2/docs/how_to/chatbots_retrieval)
    *   [How to use tools](/v0.2/docs/how_to/chatbots_tools)
    *   [How to split code](/v0.2/docs/how_to/code_splitter)
    *   [How to do retrieval with contextual compression](/v0.2/docs/how_to/contextual_compression)
    *   [How to write a custom retriever class](/v0.2/docs/how_to/custom_retriever)
    *   [How to create custom Tools](/v0.2/docs/how_to/custom_tools)
    *   [How to debug your LLM apps](/v0.2/docs/how_to/debugging)
    *   [How to load CSV data](/v0.2/docs/how_to/document_loader_csv)
    *   [How to write a custom document loader](/v0.2/docs/how_to/document_loader_custom)
    *   [How to load data from a directory](/v0.2/docs/how_to/document_loader_directory)
    *   [How to load PDF files](/v0.2/docs/how_to/document_loader_pdf)
    *   [How to load JSON data](/v0.2/docs/how_to/document_loaders_json)
    *   [How to select examples by length](/v0.2/docs/how_to/example_selectors_length_based)
    *   [How to select examples by similarity](/v0.2/docs/how_to/example_selectors_similarity)
    *   [How to use reference examples](/v0.2/docs/how_to/extraction_examples)
    *   [How to handle long text](/v0.2/docs/how_to/extraction_long_text)
    *   [How to do extraction without using function calling](/v0.2/docs/how_to/extraction_parse)
    *   [Fallbacks](/v0.2/docs/how_to/fallbacks)
    *   [Few Shot Prompt Templates](/v0.2/docs/how_to/few_shot)
    *   [How to run custom functions](/v0.2/docs/how_to/functions)
    *   [How to construct knowledge graphs](/v0.2/docs/how_to/graph_constructing)
    *   [How to map values to a database](/v0.2/docs/how_to/graph_mapping)
    *   [How to improve results with prompting](/v0.2/docs/how_to/graph_prompting)
    *   [How to add a semantic layer over the database](/v0.2/docs/how_to/graph_semantic)
    *   [How to reindex data to keep your vectorstore in-sync with the underlying data source](/v0.2/docs/how_to/indexing)
    *   [How to get log probabilities](/v0.2/docs/how_to/logprobs)
    *   [How to add message history](/v0.2/docs/how_to/message_history)
    *   [How to generate multiple embeddings per document](/v0.2/docs/how_to/multi_vector)
    *   [How to generate multiple queries to retrieve data for](/v0.2/docs/how_to/multiple_queries)
    *   [How to parse JSON output](/v0.2/docs/how_to/output_parser_json)
    *   [How to retry when output parsing errors occur](/v0.2/docs/how_to/output_parser_retry)
    *   [How to parse XML output](/v0.2/docs/how_to/output_parser_xml)
    *   [How to invoke runnables in parallel](/v0.2/docs/how_to/parallel)
    *   [How to retrieve the whole document for a chunk](/v0.2/docs/how_to/parent_document_retriever)
    *   [How to partially format prompt templates](/v0.2/docs/how_to/prompts_partial)
    *   [How to add chat history to a question-answering chain](/v0.2/docs/how_to/qa_chat_history_how_to)
    *   [How to return citations](/v0.2/docs/how_to/qa_citations)
    *   [How to return sources](/v0.2/docs/how_to/qa_sources)
    *   [How to stream from a question-answering chain](/v0.2/docs/how_to/qa_streaming)
    *   [How to construct filters](/v0.2/docs/how_to/query_constructing_filters)
    *   [How to add examples to the prompt](/v0.2/docs/how_to/query_few_shot)
    *   [How to deal with high cardinality categorical variables](/v0.2/docs/how_to/query_high_cardinality)
    *   [How to handle multiple queries](/v0.2/docs/how_to/query_multiple_queries)
    *   [How to handle multiple retrievers](/v0.2/docs/how_to/query_multiple_retrievers)
    *   [How to handle cases where no queries are generated](/v0.2/docs/how_to/query_no_queries)
    *   [How to recursively split text by characters](/v0.2/docs/how_to/recursive_text_splitter)
    *   [How to reduce retrieval latency](/v0.2/docs/how_to/reduce_retrieval_latency)
    *   [How to route execution within a chain](/v0.2/docs/how_to/routing)
    *   [How to chain runnables](/v0.2/docs/how_to/sequence)
    *   [How to split text by tokens](/v0.2/docs/how_to/split_by_token)
    *   [How to deal with large databases](/v0.2/docs/how_to/sql_large_db)
    *   [How to use prompting to improve results](/v0.2/docs/how_to/sql_prompting)
    *   [How to do query validation](/v0.2/docs/how_to/sql_query_checking)
    *   [How to stream](/v0.2/docs/how_to/streaming)
    *   [How to create a time-weighted retriever](/v0.2/docs/how_to/time_weighted_vectorstore)
    *   [How to use a chat model to call tools](/v0.2/docs/how_to/tool_calling)
    *   [How to call tools with multi-modal data](/v0.2/docs/how_to/tool_calls_multi_modal)
    *   [How to use LangChain tools](/v0.2/docs/how_to/tools_builtin)
    *   [How use a vector store to retrieve data](/v0.2/docs/how_to/vectorstore_retriever)
    *   [How to create and query vector stores](/v0.2/docs/how_to/vectorstores)
*   [Conceptual guide](/v0.2/docs/concepts)
*   Ecosystem
    
    *   [🦜🛠️ LangSmith](/v0.2/docs/langsmith/)
        
    *   [🦜🕸️LangGraph.js](/v0.2/docs/langgraph)
*   Versions
    
    *   [Overview](/v0.2/docs/versions/overview)
    *   [v0.2](/v0.2/docs/versions/v0_2)
    *   [Release Policy](/v0.2/docs/versions/release_policy)
    *   [Packages](/v0.2/docs/versions/packages)
*   [Security](/v0.2/docs/security)

*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to deal with high cardinality categorical variables

On this page

How to deal with high cardinality categorical variables
=======================================================

Prerequisites

This guide assumes familiarity with the following:

*   [Query analysis](/v0.2/docs/tutorials/query_analysis)

High cardinality data refers to columns in a dataset that contain a large number of unique values. This guide demonstrates some techniques for dealing with these inputs.

For example, you may want to do query analysis to create a filter on a categorical column. One of the difficulties here is that you usually need to specify the EXACT categorical value. The issue is you need to make sure the LLM generates that categorical value exactly. This can be done relatively easy with prompting when there are only a few values that are valid. When there are a high number of valid values then it becomes more difficult, as those values may not fit in the LLM context, or (if they do) there may be too many for the LLM to properly attend to.

In this notebook we take a look at how to approach this.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

### Install dependencies[​](#install-dependencies "Direct link to Install dependencies")

tip

See [this section for general instructions on installing integration packages](/v0.2/docs/how_to/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/community zod @faker-js/faker

    yarn add @langchain/community zod @faker-js/faker

    pnpm add @langchain/community zod @faker-js/faker

### Set environment variables[​](#set-environment-variables "Direct link to Set environment variables")

    # Optional, use LangSmith for best-in-class observabilityLANGSMITH_API_KEY=your-api-keyLANGCHAIN_TRACING_V2=true

#### Set up data[​](#set-up-data "Direct link to Set up data")

We will generate a bunch of fake names

    import { faker } from "@faker-js/faker";const names = Array.from({ length: 10000 }, () => faker.person.fullName());

Let’s look at some of the names

    names[0];

    "Rolando Wilkinson"

    names[567];

    "Homer Harber"

Query Analysis[​](#query-analysis "Direct link to Query Analysis")
------------------------------------------------------------------

We can now set up a baseline query analysis

    import { z } from "zod";const searchSchema = z.object({  query: z.string(),  author: z.string(),});

### Pick your chat model:

*   OpenAI
*   Anthropic
*   FireworksAI
*   MistralAI
*   Groq
*   VertexAI

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/openai 

    yarn add @langchain/openai 

    pnpm add @langchain/openai 

#### Add environment variables

    OPENAI_API_KEY=your-api-key

#### Instantiate the model

    import { ChatOpenAI } from "@langchain/openai";const llm = new ChatOpenAI({  model: "gpt-3.5-turbo",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/anthropic 

    yarn add @langchain/anthropic 

    pnpm add @langchain/anthropic 

#### Add environment variables

    ANTHROPIC_API_KEY=your-api-key

#### Instantiate the model

    import { ChatAnthropic } from "@langchain/anthropic";const llm = new ChatAnthropic({  model: "claude-3-sonnet-20240229",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/community 

    yarn add @langchain/community 

    pnpm add @langchain/community 

#### Add environment variables

    FIREWORKS_API_KEY=your-api-key

#### Instantiate the model

    import { ChatFireworks } from "@langchain/community/chat_models/fireworks";const llm = new ChatFireworks({  model: "accounts/fireworks/models/firefunction-v1",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/mistralai 

    yarn add @langchain/mistralai 

    pnpm add @langchain/mistralai 

#### Add environment variables

    MISTRAL_API_KEY=your-api-key

#### Instantiate the model

    import { ChatMistralAI } from "@langchain/mistralai";const llm = new ChatMistralAI({  model: "mistral-large-latest",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/groq 

    yarn add @langchain/groq 

    pnpm add @langchain/groq 

#### Add environment variables

    GROQ_API_KEY=your-api-key

#### Instantiate the model

    import { ChatGroq } from "@langchain/groq";const llm = new ChatGroq({  model: "mixtral-8x7b-32768",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/google-vertexai 

    yarn add @langchain/google-vertexai 

    pnpm add @langchain/google-vertexai 

#### Add environment variables

    GOOGLE_APPLICATION_CREDENTIALS=credentials.json

#### Instantiate the model

    import { ChatVertexAI } from "@langchain/google-vertexai";const llm = new ChatVertexAI({  model: "gemini-1.5-pro",  temperature: 0});

    import { ChatPromptTemplate } from "@langchain/core/prompts";import {  RunnablePassthrough,  RunnableSequence,} from "@langchain/core/runnables";const system = `Generate a relevant search query for a library system`;const prompt = ChatPromptTemplate.fromMessages([  ["system", system],  ["human", "{question}"],]);const llmWithTools = llm.withStructuredOutput(searchSchema, {  name: "Search",});const queryAnalyzer = RunnableSequence.from([  {    question: new RunnablePassthrough(),  },  prompt,  llmWithTools,]);

We can see that if we spell the name exactly correctly, it knows how to handle it

    await queryAnalyzer.invoke("what are books about aliens by Jesse Knight");

    { query: "aliens", author: "Jesse Knight" }

The issue is that the values you want to filter on may NOT be spelled exactly correctly

    await queryAnalyzer.invoke("what are books about aliens by jess knight");

    { query: "books about aliens", author: "jess knight" }

### Add in all values[​](#add-in-all-values "Direct link to Add in all values")

One way around this is to add ALL possible values to the prompt. That will generally guide the query in the right direction

    const system = `Generate a relevant search query for a library system using the 'search' tool.The 'author' you return to the user MUST be one of the following authors:{authors}Do NOT hallucinate author name!`;const basePrompt = ChatPromptTemplate.fromMessages([  ["system", system],  ["human", "{question}"],]);const prompt = await basePrompt.partial({ authors: names.join(", ") });const queryAnalyzerAll = RunnableSequence.from([  {    question: new RunnablePassthrough(),  },  prompt,  llmWithTools,]);

However… if the list of categoricals is long enough, it may error!

    try {  const res = await queryAnalyzerAll.invoke(    "what are books about aliens by jess knight"  );} catch (e) {  console.error(e);}

    Error: 400 This model's maximum context length is 16385 tokens. However, your messages resulted in 50197 tokens (50167 in the messages, 30 in the functions). Please reduce the length of the messages or functions.    at Function.generate (file:///Users/jacoblee/Library/Caches/deno/npm/registry.npmjs.org/openai/4.47.1/error.mjs:41:20)    at OpenAI.makeStatusError (file:///Users/jacoblee/Library/Caches/deno/npm/registry.npmjs.org/openai/4.47.1/core.mjs:256:25)    at OpenAI.makeRequest (file:///Users/jacoblee/Library/Caches/deno/npm/registry.npmjs.org/openai/4.47.1/core.mjs:299:30)    at eventLoopTick (ext:core/01_core.js:63:7)    at async file:///Users/jacoblee/Library/Caches/deno/npm/registry.npmjs.org/@langchain/openai/0.0.31/dist/chat_models.js:756:29    at async RetryOperation._fn (file:///Users/jacoblee/Library/Caches/deno/npm/registry.npmjs.org/p-retry/4.6.2/index.js:50:12) {  status: 400,  headers: {    "alt-svc": 'h3=":443"; ma=86400',    "cf-cache-status": "DYNAMIC",    "cf-ray": "885f794b3df4fa52-SJC",    "content-length": "340",    "content-type": "application/json",    date: "Sat, 18 May 2024 23:02:16 GMT",    "openai-organization": "langchain",    "openai-processing-ms": "230",    "openai-version": "2020-10-01",    server: "cloudflare",    "set-cookie": "_cfuvid=F_c9lnRuQDUhKiUE2eR2PlsxHPldf1OAVMonLlHTjzM-1716073336256-0.0.1.1-604800000; path=/; domain="... 48 more characters,    "strict-transport-security": "max-age=15724800; includeSubDomains",    "x-ratelimit-limit-requests": "10000",    "x-ratelimit-limit-tokens": "2000000",    "x-ratelimit-remaining-requests": "9999",    "x-ratelimit-remaining-tokens": "1958402",    "x-ratelimit-reset-requests": "6ms",    "x-ratelimit-reset-tokens": "1.247s",    "x-request-id": "req_7b88677d6883fac1520e44543f68c839"  },  request_id: "req_7b88677d6883fac1520e44543f68c839",  error: {    message: "This model's maximum context length is 16385 tokens. However, your messages resulted in 50197 tokens"... 101 more characters,    type: "invalid_request_error",    param: "messages",    code: "context_length_exceeded"  },  code: "context_length_exceeded",  param: "messages",  type: "invalid_request_error",  attemptNumber: 1,  retriesLeft: 6}

We can try to use a longer context window… but with so much information in there, it is not guaranteed to pick it up reliably

### Pick your chat model:

*   OpenAI
*   Anthropic
*   FireworksAI
*   MistralAI
*   Groq
*   VertexAI

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/openai 

    yarn add @langchain/openai 

    pnpm add @langchain/openai 

#### Add environment variables

    OPENAI_API_KEY=your-api-key

#### Instantiate the model

    import { ChatOpenAI } from "@langchain/openai";const llmLong = new ChatOpenAI({ model: "gpt-4-turbo-preview" });

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/anthropic 

    yarn add @langchain/anthropic 

    pnpm add @langchain/anthropic 

#### Add environment variables

    ANTHROPIC_API_KEY=your-api-key

#### Instantiate the model

    import { ChatAnthropic } from "@langchain/anthropic";const llmLong = new ChatAnthropic({  model: "claude-3-sonnet-20240229",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/community 

    yarn add @langchain/community 

    pnpm add @langchain/community 

#### Add environment variables

    FIREWORKS_API_KEY=your-api-key

#### Instantiate the model

    import { ChatFireworks } from "@langchain/community/chat_models/fireworks";const llmLong = new ChatFireworks({  model: "accounts/fireworks/models/firefunction-v1",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/mistralai 

    yarn add @langchain/mistralai 

    pnpm add @langchain/mistralai 

#### Add environment variables

    MISTRAL_API_KEY=your-api-key

#### Instantiate the model

    import { ChatMistralAI } from "@langchain/mistralai";const llmLong = new ChatMistralAI({  model: "mistral-large-latest",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/groq 

    yarn add @langchain/groq 

    pnpm add @langchain/groq 

#### Add environment variables

    GROQ_API_KEY=your-api-key

#### Instantiate the model

    import { ChatGroq } from "@langchain/groq";const llmLong = new ChatGroq({  model: "mixtral-8x7b-32768",  temperature: 0});

#### Install dependencies

tip

See [this section for general instructions on installing integration packages](/docs/get_started/installation#installing-integration-packages).

*   npm
*   yarn
*   pnpm

    npm i @langchain/google-vertexai 

    yarn add @langchain/google-vertexai 

    pnpm add @langchain/google-vertexai 

#### Add environment variables

    GOOGLE_APPLICATION_CREDENTIALS=credentials.json

#### Instantiate the model

    import { ChatVertexAI } from "@langchain/google-vertexai";const llmLong = new ChatVertexAI({  model: "gemini-1.5-pro",  temperature: 0});

    const structuredLlmLong = llmLong.withStructuredOutput(searchSchema, {  name: "Search",});const queryAnalyzerAll = RunnableSequence.from([  {    question: new RunnablePassthrough(),  },  prompt,  structuredLlmLong,]);

    await queryAnalyzerAll.invoke("what are books about aliens by jess knight");

    { query: "aliens", author: "jess knight" }

### Find and all relevant values[​](#find-and-all-relevant-values "Direct link to Find and all relevant values")

Instead, what we can do is create a [vector store index](/v0.2/docs/concepts#vectorstores) over the relevant values and then query that for the N most relevant values,

    import { OpenAIEmbeddings } from "@langchain/openai";import { MemoryVectorStore } from "langchain/vectorstores/memory";const embeddings = new OpenAIEmbeddings({  model: "text-embedding-3-small",});const vectorstore = await MemoryVectorStore.fromTexts(names, {}, embeddings);const selectNames = async (question: string) => {  const _docs = await vectorstore.similaritySearch(question, 10);  const _names = _docs.map((d) => d.pageContent);  return _names.join(", ");};const createPrompt = RunnableSequence.from([  {    question: new RunnablePassthrough(),    authors: selectNames,  },  basePrompt,]);await createPrompt.invoke("what are books by jess knight");

    ChatPromptValue {  lc_serializable: true,  lc_kwargs: {    messages: [      SystemMessage {        lc_serializable: true,        lc_kwargs: {          content: "Generate a relevant search query for a library system using the 'search' tool.\n" +            "\n" +            "The 'author' you ret"... 243 more characters,          additional_kwargs: {},          response_metadata: {}        },        lc_namespace: [ "langchain_core", "messages" ],        content: "Generate a relevant search query for a library system using the 'search' tool.\n" +          "\n" +          "The 'author' you ret"... 243 more characters,        name: undefined,        additional_kwargs: {},        response_metadata: {}      },      HumanMessage {        lc_serializable: true,        lc_kwargs: {          content: "what are books by jess knight",          additional_kwargs: {},          response_metadata: {}        },        lc_namespace: [ "langchain_core", "messages" ],        content: "what are books by jess knight",        name: undefined,        additional_kwargs: {},        response_metadata: {}      }    ]  },  lc_namespace: [ "langchain_core", "prompt_values" ],  messages: [    SystemMessage {      lc_serializable: true,      lc_kwargs: {        content: "Generate a relevant search query for a library system using the 'search' tool.\n" +          "\n" +          "The 'author' you ret"... 243 more characters,        additional_kwargs: {},        response_metadata: {}      },      lc_namespace: [ "langchain_core", "messages" ],      content: "Generate a relevant search query for a library system using the 'search' tool.\n" +        "\n" +        "The 'author' you ret"... 243 more characters,      name: undefined,      additional_kwargs: {},      response_metadata: {}    },    HumanMessage {      lc_serializable: true,      lc_kwargs: {        content: "what are books by jess knight",        additional_kwargs: {},        response_metadata: {}      },      lc_namespace: [ "langchain_core", "messages" ],      content: "what are books by jess knight",      name: undefined,      additional_kwargs: {},      response_metadata: {}    }  ]}

    const queryAnalyzerSelect = createPrompt.pipe(llmWithTools);await queryAnalyzerSelect.invoke("what are books about aliens by jess knight");

    { query: "aliens", author: "Jess Knight" }

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned how to deal with high cardinality data when constructing queries.

Next, check out some of the other query analysis guides in this section, like [how to use few-shotting to improve performance](/v0.2/docs/how_to/query_no_queries).

* * *

#### Was this page helpful?

  

#### You can leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to add examples to the prompt

](/v0.2/docs/how_to/query_few_shot)[

Next

How to handle multiple queries

](/v0.2/docs/how_to/query_multiple_queries)

*   [Setup](#setup)
    *   [Install dependencies](#install-dependencies)
    *   [Set environment variables](#set-environment-variables)
*   [Query Analysis](#query-analysis)
    *   [Add in all values](#add-in-all-values)
    *   [Find and all relevant values](#find-and-all-relevant-values)
*   [Next steps](#next-steps)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.