*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to cache chat model responses

On this page

How to cache chat model responses
=================================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Chat models](/v0.2/docs/concepts/#chat-models)
*   [LLMs](/v0.2/docs/concepts/#llms)

LangChain provides an optional caching layer for chat models. This is useful for two main reasons:

*   It can save you money by reducing the number of API calls you make to the LLM provider, if you're often requesting the same completion multiple times. This is especially useful during app development.
*   It can speed up your application by reducing the number of API calls you make to the LLM provider.

This guide will walk you through how to enable this in your apps.

*   OpenAI
*   Anthropic
*   Azure
*   Google
*   Cohere
*   FireworksAI
*   Groq
*   MistralAI
*   TogetherAI

    pip install -qU langchain-openai

    import getpassimport osos.environ["OPENAI_API_KEY"] = getpass.getpass()from langchain_openai import ChatOpenAIllm = ChatOpenAI(model="gpt-3.5-turbo-0125")

    pip install -qU langchain-anthropic

    import getpassimport osos.environ["ANTHROPIC_API_KEY"] = getpass.getpass()from langchain_anthropic import ChatAnthropicllm = ChatAnthropic(model="claude-3-sonnet-20240229")

    pip install -qU langchain-openai

    import getpassimport osos.environ["AZURE_OPENAI_API_KEY"] = getpass.getpass()from langchain_openai import AzureChatOpenAIllm = AzureChatOpenAI(    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],    azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"],    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],)

    pip install -qU langchain-google-vertexai

    import getpassimport osos.environ["GOOGLE_API_KEY"] = getpass.getpass()from langchain_google_vertexai import ChatVertexAIllm = ChatVertexAI(model="gemini-pro")

    pip install -qU langchain-cohere

    import getpassimport osos.environ["COHERE_API_KEY"] = getpass.getpass()from langchain_cohere import ChatCoherellm = ChatCohere(model="command-r")

    pip install -qU langchain-fireworks

    import getpassimport osos.environ["FIREWORKS_API_KEY"] = getpass.getpass()from langchain_fireworks import ChatFireworksllm = ChatFireworks(model="accounts/fireworks/models/mixtral-8x7b-instruct")

    pip install -qU langchain-groq

    import getpassimport osos.environ["GROQ_API_KEY"] = getpass.getpass()from langchain_groq import ChatGroqllm = ChatGroq(model="llama3-8b-8192")

    pip install -qU langchain-mistralai

    import getpassimport osos.environ["MISTRAL_API_KEY"] = getpass.getpass()from langchain_mistralai import ChatMistralAIllm = ChatMistralAI(model="mistral-large-latest")

    pip install -qU langchain-openai

    import getpassimport osos.environ["TOGETHER_API_KEY"] = getpass.getpass()from langchain_openai import ChatOpenAIllm = ChatOpenAI(    base_url="https://api.together.xyz/v1",    api_key=os.environ["TOGETHER_API_KEY"],    model="mistralai/Mixtral-8x7B-Instruct-v0.1",)

    # <!-- ruff: noqa: F821 -->from langchain.globals import set_llm_cache

**API Reference:**[set\_llm\_cache](https://api.python.langchain.com/en/latest/globals/langchain.globals.set_llm_cache.html)

In Memory Cache[​](#in-memory-cache "Direct link to In Memory Cache")
---------------------------------------------------------------------

This is an ephemeral cache that stores model calls in memory. It will be wiped when your environment restarts, and is not shared across processes.

    %%timefrom langchain.cache import InMemoryCacheset_llm_cache(InMemoryCache())# The first time, it is not yet in cache, so it should take longerllm.invoke("Tell me a joke")

**API Reference:**[InMemoryCache](https://api.python.langchain.com/en/latest/cache/langchain_community.cache.InMemoryCache.html)

    CPU times: user 645 ms, sys: 214 ms, total: 859 msWall time: 829 ms

    AIMessage(content="Why don't scientists trust atoms?\n\nBecause they make up everything!", response_metadata={'token_usage': {'completion_tokens': 13, 'prompt_tokens': 11, 'total_tokens': 24}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_c2295e73ad', 'finish_reason': 'stop', 'logprobs': None}, id='run-b6836bdd-8c30-436b-828f-0ac5fc9ab50e-0')

    %%time# The second time it is, so it goes fasterllm.invoke("Tell me a joke")

    CPU times: user 822 µs, sys: 288 µs, total: 1.11 msWall time: 1.06 ms

    AIMessage(content="Why don't scientists trust atoms?\n\nBecause they make up everything!", response_metadata={'token_usage': {'completion_tokens': 13, 'prompt_tokens': 11, 'total_tokens': 24}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_c2295e73ad', 'finish_reason': 'stop', 'logprobs': None}, id='run-b6836bdd-8c30-436b-828f-0ac5fc9ab50e-0')

SQLite Cache[​](#sqlite-cache "Direct link to SQLite Cache")
------------------------------------------------------------

This cache implementation uses a `SQLite` database to store responses, and will last across process restarts.

    !rm .langchain.db

    # We can do the same thing with a SQLite cachefrom langchain_community.cache import SQLiteCacheset_llm_cache(SQLiteCache(database_path=".langchain.db"))

**API Reference:**[SQLiteCache](https://api.python.langchain.com/en/latest/cache/langchain_community.cache.SQLiteCache.html)

    %%time# The first time, it is not yet in cache, so it should take longerllm.invoke("Tell me a joke")

    CPU times: user 9.91 ms, sys: 7.68 ms, total: 17.6 msWall time: 657 ms

    AIMessage(content='Why did the scarecrow win an award? Because he was outstanding in his field!', response_metadata={'token_usage': {'completion_tokens': 17, 'prompt_tokens': 11, 'total_tokens': 28}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_c2295e73ad', 'finish_reason': 'stop', 'logprobs': None}, id='run-39d9e1e8-7766-4970-b1d8-f50213fd94c5-0')

    %%time# The second time it is, so it goes fasterllm.invoke("Tell me a joke")

    CPU times: user 52.2 ms, sys: 60.5 ms, total: 113 msWall time: 127 ms

    AIMessage(content='Why did the scarecrow win an award? Because he was outstanding in his field!', id='run-39d9e1e8-7766-4970-b1d8-f50213fd94c5-0')

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You've now learned how to cache model responses to save time and money.

Next, check out the other how-to guides chat models in this section, like [how to get a model to return structured output](/v0.2/docs/how_to/structured_output/) or [how to create your own custom chat model](/v0.2/docs/how_to/custom_chat_model/).

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/how_to/chat_model_caching.ipynb)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to split by character

](/v0.2/docs/how_to/character_text_splitter/)[

Next

How to init any model in one line

](/v0.2/docs/how_to/chat_models_universal_init/)

*   [In Memory Cache](#in-memory-cache)
*   [SQLite Cache](#sqlite-cache)
*   [Next steps](#next-steps)