!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.1/docs/integrations/platforms/)
    
    *   [Providers](/v0.1/docs/integrations/platforms/)
    *   [Anthropic](/v0.1/docs/integrations/platforms/anthropic/)
    *   [AWS](/v0.1/docs/integrations/platforms/aws/)
    *   [Google](/v0.1/docs/integrations/platforms/google/)
    *   [Microsoft](/v0.1/docs/integrations/platforms/microsoft/)
    *   [OpenAI](/v0.1/docs/integrations/platforms/openai/)
*   [Components](/v0.1/docs/integrations/components/)
    
    *   [LLMs](/v0.1/docs/integrations/llms/)
        
    *   [Chat models](/v0.1/docs/integrations/chat/)
        
        *   [Chat models](/v0.1/docs/integrations/chat/)
        *   [Alibaba Tongyi](/v0.1/docs/integrations/chat/alibaba_tongyi/)
        *   [Anthropic](/v0.1/docs/integrations/chat/anthropic/)
        *   [Anthropic Tools](/v0.1/docs/integrations/chat/anthropic_tools/)
        *   [Azure OpenAI](/v0.1/docs/integrations/chat/azure/)
        *   [Baidu Wenxin](/v0.1/docs/integrations/chat/baidu_wenxin/)
        *   [Bedrock](/v0.1/docs/integrations/chat/bedrock/)
        *   [Cloudflare Workers AI](/v0.1/docs/integrations/chat/cloudflare_workersai/)
        *   [Cohere](/v0.1/docs/integrations/chat/cohere/)
        *   [Fake LLM](/v0.1/docs/integrations/chat/fake/)
        *   [Fireworks](/v0.1/docs/integrations/chat/fireworks/)
        *   [Friendli](/v0.1/docs/integrations/chat/friendli/)
        *   [Google GenAI](/v0.1/docs/integrations/chat/google_generativeai/)
        *   [(Legacy) Google PaLM/VertexAI](/v0.1/docs/integrations/chat/google_palm/)
        *   [Google Vertex AI](/v0.1/docs/integrations/chat/google_vertex_ai/)
        *   [Groq](/v0.1/docs/integrations/chat/groq/)
        *   [Llama CPP](/v0.1/docs/integrations/chat/llama_cpp/)
        *   [Minimax](/v0.1/docs/integrations/chat/minimax/)
        *   [Mistral AI](/v0.1/docs/integrations/chat/mistral/)
        *   [NIBittensorChatModel](/v0.1/docs/integrations/chat/ni_bittensor/)
        *   [Ollama](/v0.1/docs/integrations/chat/ollama/)
        *   [Ollama Functions](/v0.1/docs/integrations/chat/ollama_functions/)
        *   [OpenAI](/v0.1/docs/integrations/chat/openai/)
        *   [PremAI](/v0.1/docs/integrations/chat/premai/)
        *   [PromptLayer OpenAI](/v0.1/docs/integrations/chat/prompt_layer_openai/)
        *   [TogetherAI](/v0.1/docs/integrations/chat/togetherai/)
        *   [WebLLM](/v0.1/docs/integrations/chat/web_llm/)
        *   [YandexGPT](/v0.1/docs/integrations/chat/yandex/)
        *   [ZhipuAI](/v0.1/docs/integrations/chat/zhipuai/)
    *   [Document loaders](/v0.1/docs/integrations/document_loaders/)
        
    *   [Document transformers](/v0.1/docs/integrations/document_transformers/)
        
    *   [Document compressors](/v0.1/docs/integrations/document_compressors/)
        
    *   [Text embedding models](/v0.1/docs/integrations/text_embedding/)
        
    *   [Vector stores](/v0.1/docs/integrations/vectorstores/)
        
    *   [Retrievers](/v0.1/docs/integrations/retrievers/)
        
    *   [Tools](/v0.1/docs/integrations/tools/)
        
    *   [Agents and toolkits](/v0.1/docs/integrations/toolkits/)
        
    *   [Chat Memory](/v0.1/docs/integrations/chat_memory/)
        
    *   [Stores](/v0.1/docs/integrations/stores/)
        

*   [](/v0.1/)
*   [Components](/v0.1/docs/integrations/components/)
*   [Chat models](/v0.1/docs/integrations/chat/)
*   Llama CPP

On this page

Llama CPP
=========

Compatibility

Only available on Node.js.

This module is based on the [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) Node.js bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp), allowing you to work with a locally running LLM. This allows you to work with a much smaller quantized model capable of running on a laptop environment, ideal for testing and scratch padding ideas without running up a bill!

Setup[​](#setup "Direct link to Setup")
---------------------------------------

You'll need to install the [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) module to communicate with your local model.

tip

See [this section for general instructions on installing integration packages](/v0.1/docs/get_started/installation/#installing-integration-packages).

*   npm
*   Yarn
*   pnpm

    npm install -S node-llama-cpp @langchain/community

    yarn add node-llama-cpp @langchain/community

    pnpm add node-llama-cpp @langchain/community

You will also need a local Llama 2 model (or a model supported by [node-llama-cpp](https://github.com/withcatai/node-llama-cpp)). You will need to pass the path to this model to the LlamaCpp module as a part of the parameters (see example).

Out-of-the-box `node-llama-cpp` is tuned for running on a MacOS platform with support for the Metal GPU of Apple M-series of processors. If you need to turn this off or need support for the CUDA architecture then refer to the documentation at [node-llama-cpp](https://withcatai.github.io/node-llama-cpp/).

For advice on getting and preparing `llama2` see the documentation for the LLM version of this module.

A note to LangChain.js contributors: if you want to run the tests associated with this module you will need to put the path to your local model in the environment variable `LLAMA_PATH`.

Usage[​](#usage "Direct link to Usage")
---------------------------------------

### Basic use[​](#basic-use "Direct link to Basic use")

In this case we pass in a prompt wrapped as a message and expect a response.

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";import { HumanMessage } from "@langchain/core/messages";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new ChatLlamaCpp({ modelPath: llamaPath });const response = await model.invoke([  new HumanMessage({ content: "My name is John." }),]);console.log({ response });/*  AIMessage {    lc_serializable: true,    lc_kwargs: {      content: 'Hello John.',      additional_kwargs: {}    },    lc_namespace: [ 'langchain', 'schema' ],    content: 'Hello John.',    name: undefined,    additional_kwargs: {}  }*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`
*   [HumanMessage](https://api.js.langchain.com/classes/langchain_core_messages.HumanMessage.html) from `@langchain/core/messages`

### System messages[​](#system-messages "Direct link to System messages")

We can also provide a system message, note that with the `llama_cpp` module a system message will cause the creation of a new session.

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";import { SystemMessage, HumanMessage } from "@langchain/core/messages";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new ChatLlamaCpp({ modelPath: llamaPath });const response = await model.invoke([  new SystemMessage(    "You are a pirate, responses must be very verbose and in pirate dialect, add 'Arr, m'hearty!' to each sentence."  ),  new HumanMessage("Tell me where Llamas come from?"),]);console.log({ response });/*  AIMessage {    lc_serializable: true,    lc_kwargs: {      content: "Arr, m'hearty! Llamas come from the land of Peru.",      additional_kwargs: {}    },    lc_namespace: [ 'langchain', 'schema' ],    content: "Arr, m'hearty! Llamas come from the land of Peru.",    name: undefined,    additional_kwargs: {}  }*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`
*   [SystemMessage](https://api.js.langchain.com/classes/langchain_core_messages.SystemMessage.html) from `@langchain/core/messages`
*   [HumanMessage](https://api.js.langchain.com/classes/langchain_core_messages.HumanMessage.html) from `@langchain/core/messages`

### Chains[​](#chains "Direct link to Chains")

This module can also be used with chains, note that using more complex chains will require suitably powerful version of `llama2` such as the 70B version.

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";import { LLMChain } from "langchain/chains";import { PromptTemplate } from "@langchain/core/prompts";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new ChatLlamaCpp({ modelPath: llamaPath, temperature: 0.5 });const prompt = PromptTemplate.fromTemplate(  "What is a good name for a company that makes {product}?");const chain = new LLMChain({ llm: model, prompt });const response = await chain.invoke({ product: "colorful socks" });console.log({ response });/*  {  text: `I'm not sure what you mean by "colorful socks" but here are some ideas:\n` +    '\n' +    '- Sock-it to me!\n' +    '- Socks Away\n' +    '- Fancy Footwear'  }*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`
*   [LLMChain](https://api.js.langchain.com/classes/langchain_chains.LLMChain.html) from `langchain/chains`
*   [PromptTemplate](https://api.js.langchain.com/classes/langchain_core_prompts.PromptTemplate.html) from `@langchain/core/prompts`

### Streaming[​](#streaming "Direct link to Streaming")

We can also stream with Llama CPP, this can be using a raw 'single prompt' string:

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new ChatLlamaCpp({ modelPath: llamaPath, temperature: 0.7 });const stream = await model.stream("Tell me a short story about a happy Llama.");for await (const chunk of stream) {  console.log(chunk.content);}/*  Once   upon   a   time  ,   in   a   green   and   sunny   field  ...*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`

Or you can provide multiple messages, note that this takes the input and then submits a Llama2 formatted prompt to the model.

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";import { SystemMessage, HumanMessage } from "@langchain/core/messages";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const llamaCpp = new ChatLlamaCpp({ modelPath: llamaPath, temperature: 0.7 });const stream = await llamaCpp.stream([  new SystemMessage(    "You are a pirate, responses must be very verbose and in pirate dialect."  ),  new HumanMessage("Tell me about Llamas?"),]);for await (const chunk of stream) {  console.log(chunk.content);}/*  Ar  rr  r  ,   me   heart  y  !   Ye   be   ask  in  '   about   llam  as  ,   e  h  ?  ...*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`
*   [SystemMessage](https://api.js.langchain.com/classes/langchain_core_messages.SystemMessage.html) from `@langchain/core/messages`
*   [HumanMessage](https://api.js.langchain.com/classes/langchain_core_messages.HumanMessage.html) from `@langchain/core/messages`

Using the `invoke` method, we can also achieve stream generation, and use `signal` to abort the generation.

    import { ChatLlamaCpp } from "@langchain/community/chat_models/llama_cpp";import { SystemMessage, HumanMessage } from "@langchain/core/messages";const llamaPath = "/Replace/with/path/to/your/model/gguf-llama2-q4_0.bin";const model = new ChatLlamaCpp({ modelPath: llamaPath, temperature: 0.7 });const controller = new AbortController();setTimeout(() => {  controller.abort();  console.log("Aborted");}, 5000);await model.invoke(  [    new SystemMessage(      "You are a pirate, responses must be very verbose and in pirate dialect."    ),    new HumanMessage("Tell me about Llamas?"),  ],  {    signal: controller.signal,    callbacks: [      {        handleLLMNewToken(token) {          console.log(token);        },      },    ],  });/*  Once   upon   a   time  ,   in   a   green   and   sunny   field  ...  Aborted  AbortError*/

#### API Reference:

*   [ChatLlamaCpp](https://api.js.langchain.com/classes/langchain_community_chat_models_llama_cpp.ChatLlamaCpp.html) from `@langchain/community/chat_models/llama_cpp`
*   [SystemMessage](https://api.js.langchain.com/classes/langchain_core_messages.SystemMessage.html) from `@langchain/core/messages`
*   [HumanMessage](https://api.js.langchain.com/classes/langchain_core_messages.HumanMessage.html) from `@langchain/core/messages`

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

Groq

](/v0.1/docs/integrations/chat/groq/)[

Next

Minimax

](/v0.1/docs/integrations/chat/minimax/)

*   [Setup](#setup)
*   [Usage](#usage)
    *   [Basic use](#basic-use)
    *   [System messages](#system-messages)
    *   [Chains](#chains)
    *   [Streaming](#streaming)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.