seansullivan/Langchain-Docs huggingface dataset:
## Cohere

> [Cohere](https://cohere.ai/about) is a Canadian startup that provides natural language processing models that help companies improve human-machine interactions.

Head to the [API reference](https://api.python.langchain.com/en/latest/llms/langchain_community.llms.cohere.Cohere.html) for detailed documentation of all attributes and methods.

## Setup[​](#setup "Direct link to Setup")

The integration lives in the `langchain-community` package. We also need to install the `cohere` package itself. We can install these with:

```
pip install -U langchain-community langchain-cohere
```

We’ll also need to get a [Cohere API key](https://cohere.com/) and set the `COHERE_API_KEY` environment variable:

```
import getpassimport osos.environ["COHERE_API_KEY"] = getpass.getpass()
```

It’s also helpful (but not needed) to set up [LangSmith](https://smith.langchain.com/) for best-in-class observability

```
# os.environ["LANGCHAIN_TRACING_V2"] = "true"# os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()
```

## Usage[​](#usage "Direct link to Usage")

Cohere supports all [LLM](https://python.langchain.com/docs/modules/model_io/llms/) functionality:

```
from langchain_cohere import Coherefrom langchain_core.messages import HumanMessage
```

```
model = Cohere(model="command", max_tokens=256, temperature=0.75)
```

```
message = "Knock knock"model.invoke(message)
```

```
await model.ainvoke(message)
```

```
for chunk in model.stream(message):    print(chunk, end="", flush=True)
```

You can also easily combine with a prompt template for easy structuring of user input. We can do this using [LCEL](https://python.langchain.com/docs/expression_language/)

```
from langchain_core.prompts import PromptTemplateprompt = PromptTemplate.from_template("Tell me a joke about {topic}")chain = prompt | model
```

```
chain.invoke({"topic": "bears"})
```

```
' Why did the teddy bear cross the road?\nBecause he had bear crossings.\n\nWould you like to hear another joke? '
```
## C Transformers

The [C Transformers](https://github.com/marella/ctransformers) library provides Python bindings for GGML models.

This example goes over how to use LangChain to interact with `C Transformers` [models](https://github.com/marella/ctransformers#supported-models).

**Install**

```
%pip install --upgrade --quiet  ctransformers
```

**Load Model**

```
from langchain_community.llms import CTransformersllm = CTransformers(model="marella/gpt-2-ggml")
```

**Generate Text**

```
print(llm("AI is going to"))
```

**Streaming**

```
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandlerllm = CTransformers(    model="marella/gpt-2-ggml", callbacks=[StreamingStdOutCallbackHandler()])response = llm("AI is going to")
```

**LLMChain**

```
from langchain.chains import LLMChainfrom langchain_core.prompts import PromptTemplatetemplate = """Question: {question}Answer:"""prompt = PromptTemplate.from_template(template)llm_chain = LLMChain(prompt=prompt, llm=llm)response = llm_chain.run("What is AI?")
```

* * *

#### Help us out by providing feedback on this documentation page:
## ForefrontAI

The `Forefront` platform gives you the ability to fine-tune and use [open-source large language models](https://docs.forefront.ai/forefront/master/models).

This notebook goes over how to use Langchain with [ForefrontAI](https://www.forefront.ai/).

## Imports[​](#imports "Direct link to Imports")

```
import osfrom langchain.chains import LLMChainfrom langchain_community.llms import ForefrontAIfrom langchain_core.prompts import PromptTemplate
```

## Set the Environment API Key[​](#set-the-environment-api-key "Direct link to Set the Environment API Key")

Make sure to get your API key from ForefrontAI. You are given a 5 day free trial to test different models.

```
# get a new token: https://docs.forefront.ai/forefront/api-reference/authenticationfrom getpass import getpassFOREFRONTAI_API_KEY = getpass()
```

```
os.environ["FOREFRONTAI_API_KEY"] = FOREFRONTAI_API_KEY
```

## Create the ForefrontAI instance[​](#create-the-forefrontai-instance "Direct link to Create the ForefrontAI instance")

You can specify different parameters such as the model endpoint url, length, temperature, etc. You must provide an endpoint url.

```
llm = ForefrontAI(endpoint_url="YOUR ENDPOINT URL HERE")
```

## Create a Prompt Template[​](#create-a-prompt-template "Direct link to Create a Prompt Template")

We will create a prompt template for Question and Answer.

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

## Initiate the LLMChain[​](#initiate-the-llmchain "Direct link to Initiate the LLMChain")

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

## Run the LLMChain[​](#run-the-llmchain "Direct link to Run the LLMChain")

Provide a question and run the LLMChain.

```
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)
```
## GigaChat

This notebook shows how to use LangChain with [GigaChat](https://developers.sber.ru/portal/products/gigachat). To use you need to install `gigachat` python package.

```
%pip install --upgrade --quiet  gigachat
```

To get GigaChat credentials you need to [create account](https://developers.sber.ru/studio/login) and [get access to API](https://developers.sber.ru/docs/ru/gigachat/individuals-quickstart)

## Example[​](#example "Direct link to Example")

```
import osfrom getpass import getpassos.environ["GIGACHAT_CREDENTIALS"] = getpass()
```

```
from langchain_community.llms import GigaChatllm = GigaChat(verify_ssl_certs=False, scope="GIGACHAT_API_PERS")
```

```
from langchain.chains import LLMChainfrom langchain_core.prompts import PromptTemplatetemplate = "What is capital of {country}?"prompt = PromptTemplate.from_template(template)llm_chain = LLMChain(prompt=prompt, llm=llm)generated = llm_chain.invoke(input={"country": "Russia"})print(generated["text"])
```

```
The capital of Russia is Moscow.
```

* * *

#### Help us out by providing feedback on this documentation page:
## DeepInfra

[DeepInfra](https://deepinfra.com/?utm_source=langchain) is a serverless inference as a service that provides access to a [variety of LLMs](https://deepinfra.com/models?utm_source=langchain) and [embeddings models](https://deepinfra.com/models?type=embeddings&utm_source=langchain). This notebook goes over how to use LangChain with DeepInfra for language models.

## Set the Environment API Key[​](#set-the-environment-api-key "Direct link to Set the Environment API Key")

Make sure to get your API key from DeepInfra. You have to [Login](https://deepinfra.com/login?from=%2Fdash) and get a new token.

You are given a 1 hour free of serverless GPU compute to test different models. (see [here](https://github.com/deepinfra/deepctl#deepctl)) You can print your token with `deepctl auth token`

```
# get a new token: https://deepinfra.com/login?from=%2Fdashfrom getpass import getpassDEEPINFRA_API_TOKEN = getpass()
```

```
import osos.environ["DEEPINFRA_API_TOKEN"] = DEEPINFRA_API_TOKEN
```

## Create the DeepInfra instance[​](#create-the-deepinfra-instance "Direct link to Create the DeepInfra instance")

You can also use our open-source [deepctl tool](https://github.com/deepinfra/deepctl#deepctl) to manage your model deployments. You can view a list of available parameters [here](https://deepinfra.com/databricks/dolly-v2-12b#API).

```
from langchain_community.llms import DeepInfrallm = DeepInfra(model_id="meta-llama/Llama-2-70b-chat-hf")llm.model_kwargs = {    "temperature": 0.7,    "repetition_penalty": 1.2,    "max_new_tokens": 250,    "top_p": 0.9,}
```

```
# run inferences directly via wrapperllm("Who let the dogs out?")
```

```
'This is a question that has puzzled many people'
```

```
# run streaming inferencefor chunk in llm.stream("Who let the dogs out?"):    print(chunk)
```

## Create a Prompt Template[​](#create-a-prompt-template "Direct link to Create a Prompt Template")

We will create a prompt template for Question and Answer.

```
from langchain_core.prompts import PromptTemplatetemplate = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

## Initiate the LLMChain[​](#initiate-the-llmchain "Direct link to Initiate the LLMChain")

```
from langchain.chains import LLMChainllm_chain = LLMChain(prompt=prompt, llm=llm)
```

## Run the LLMChain[​](#run-the-llmchain "Direct link to Run the LLMChain")

Provide a question and run the LLMChain.

```
question = "Can penguins reach the North pole?"llm_chain.run(question)
```

```
"Penguins are found in Antarctica and the surrounding islands, which are located at the southernmost tip of the planet. The North Pole is located at the northernmost tip of the planet, and it would be a long journey for penguins to get there. In fact, penguins don't have the ability to fly or migrate over such long distances. So, no, penguins cannot reach the North Pole. "
```
This page covers how to use the [DeepSparse](https://github.com/neuralmagic/deepsparse) inference runtime within LangChain. It is broken into two parts: installation and setup, and then examples of DeepSparse usage.

There exists a DeepSparse LLM wrapper, that provides a unified interface for all models:

```
config = {"max_generated_tokens": 256}llm = DeepSparse(    model="zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none",    config=config,)
```
## GooseAI

`GooseAI` is a fully managed NLP-as-a-Service, delivered via API. GooseAI provides access to [these models](https://goose.ai/docs/models).

This notebook goes over how to use Langchain with [GooseAI](https://goose.ai/).

## Install openai[​](#install-openai "Direct link to Install openai")

The `openai` package is required to use the GooseAI API. Install `openai` using `pip install openai`.

```
%pip install --upgrade --quiet  langchain-openai
```

## Imports[​](#imports "Direct link to Imports")

```
import osfrom langchain.chains import LLMChainfrom langchain_community.llms import GooseAIfrom langchain_core.prompts import PromptTemplate
```

## Set the Environment API Key[​](#set-the-environment-api-key "Direct link to Set the Environment API Key")

Make sure to get your API key from GooseAI. You are given \\$10 in free credits to test different models.

```
from getpass import getpassGOOSEAI_API_KEY = getpass()
```

```
os.environ["GOOSEAI_API_KEY"] = GOOSEAI_API_KEY
```

## Create the GooseAI instance[​](#create-the-gooseai-instance "Direct link to Create the GooseAI instance")

You can specify different parameters such as the model name, max tokens generated, temperature, etc.

## Create a Prompt Template[​](#create-a-prompt-template "Direct link to Create a Prompt Template")

We will create a prompt template for Question and Answer.

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

## Initiate the LLMChain[​](#initiate-the-llmchain "Direct link to Initiate the LLMChain")

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

## Run the LLMChain[​](#run-the-llmchain "Direct link to Run the LLMChain")

Provide a question and run the LLMChain.

```
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)
```

* * *

#### Help us out by providing feedback on this documentation page:
## GPT4All

[GitHub:nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all) an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue.

This example goes over how to use LangChain to interact with `GPT4All` models.

```
%pip install --upgrade --quiet  gpt4all > /dev/null
```

```
Note: you may need to restart the kernel to use updated packages.
```

### Import GPT4All[​](#import-gpt4all "Direct link to Import GPT4All")

```
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandlerfrom langchain.chains import LLMChainfrom langchain_community.llms import GPT4Allfrom langchain_core.prompts import PromptTemplate
```

### Set Up Question to pass to LLM[​](#set-up-question-to-pass-to-llm "Direct link to Set Up Question to pass to LLM")

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

### Specify Model[​](#specify-model "Direct link to Specify Model")

To run locally, download a compatible ggml-formatted model.

The [gpt4all page](https://gpt4all.io/index.html) has a useful `Model Explorer` section:

*   Select a model of interest
*   Download using the UI and move the `.bin` to the `local_path` (noted below)

For more info, visit [https://github.com/nomic-ai/gpt4all](https://github.com/nomic-ai/gpt4all).

* * *

```
local_path = (    "./models/ggml-gpt4all-l13b-snoozy.bin"  # replace with your desired local file path)
```

```
# Callbacks support token-wise streamingcallbacks = [StreamingStdOutCallbackHandler()]# Verbose is required to pass to the callback managerllm = GPT4All(model=local_path, callbacks=callbacks, verbose=True)# If you want to use a custom model add the backend parameter# Check https://docs.gpt4all.io/gpt4all_python.html for supported backendsllm = GPT4All(model=local_path, backend="gptj", callbacks=callbacks, verbose=True)
```

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

```
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"llm_chain.run(question)
```

Justin Bieber was born on March 1, 1994. In 1994, The Cowboys won Super Bowl XXVIII.
[KoboldAI](https://github.com/KoboldAI/KoboldAI-Client) is a “a browser-based front-end for AI-assisted writing with multiple local & remote AI models…”. It has a public and local API that is able to be used in langchain.

This example goes over how to use LangChain with that API.

Documentation can be found in the browser adding /api to the end of your endpoint (i.e [http://127.0.0.1/:5000/api](http://127.0.0.1/:5000/api)).

Replace the endpoint seen below with the one shown in the output after starting the webui with –api or –public-api

```
llm = KoboldApiLLM(endpoint="http://192.168.1.144:5000", max_length=80)
```
## PromptLayer OpenAI

`PromptLayer` is the first platform that allows you to track, manage, and share your GPT prompt engineering. `PromptLayer` acts a middleware between your code and `OpenAI’s` python library.

`PromptLayer` records all your `OpenAI API` requests, allowing you to search and explore request history in the `PromptLayer` dashboard.

This example showcases how to connect to [PromptLayer](https://www.promptlayer.com/) to start recording your OpenAI requests.

Another example is [here](https://python.langchain.com/docs/integrations/providers/promptlayer/).

## Install PromptLayer[​](#install-promptlayer "Direct link to Install PromptLayer")

The `promptlayer` package is required to use PromptLayer with OpenAI. Install `promptlayer` using pip.

```
%pip install --upgrade --quiet  promptlayer
```

## Imports[​](#imports "Direct link to Imports")

```
import osimport promptlayerfrom langchain_community.llms import PromptLayerOpenAI
```

## Set the Environment API Key[​](#set-the-environment-api-key "Direct link to Set the Environment API Key")

You can create a PromptLayer API Key at [www.promptlayer.com](https://www.promptlayer.com/) by clicking the settings cog in the navbar.

Set it as an environment variable called `PROMPTLAYER_API_KEY`.

You also need an OpenAI Key, called `OPENAI_API_KEY`.

```
from getpass import getpassPROMPTLAYER_API_KEY = getpass()
```

```
os.environ["PROMPTLAYER_API_KEY"] = PROMPTLAYER_API_KEY
```

```
from getpass import getpassOPENAI_API_KEY = getpass()
```

```
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
```

## Use the PromptLayerOpenAI LLM like normal[​](#use-the-promptlayeropenai-llm-like-normal "Direct link to Use the PromptLayerOpenAI LLM like normal")

_You can optionally pass in `pl_tags` to track your requests with PromptLayer’s tagging feature._

```
llm = PromptLayerOpenAI(pl_tags=["langchain"])llm("I am a cat and I want")
```

**The above request should now appear on your [PromptLayer dashboard](https://www.promptlayer.com/).**

## Using PromptLayer Track[​](#using-promptlayer-track "Direct link to Using PromptLayer Track")

If you would like to use any of the [PromptLayer tracking features](https://magniv.notion.site/Track-4deee1b1f7a34c1680d085f82567dab9), you need to pass the argument `return_pl_id` when instantiating the PromptLayer LLM to get the request id.

```
llm = PromptLayerOpenAI(return_pl_id=True)llm_results = llm.generate(["Tell me a joke"])for res in llm_results.generations:    pl_request_id = res[0].generation_info["pl_request_id"]    promptlayer.track.score(request_id=pl_request_id, score=100)
```

Using this allows you to track the performance of your model in the PromptLayer dashboard. If you are using a prompt template, you can attach a template to a request as well. Overall, this gives you the opportunity to track the performance of different templates and models in the PromptLayer dashboard.
## Prediction Guard

```
%pip install --upgrade --quiet  predictionguard langchain
```

```
import osfrom langchain.chains import LLMChainfrom langchain_community.llms import PredictionGuardfrom langchain_core.prompts import PromptTemplate
```

## Basic LLM usage[​](#basic-llm-usage "Direct link to Basic LLM usage")

```
# Optional, add your OpenAI API Key. This is optional, as Prediction Guard allows# you to access all the latest open access models (see https://docs.predictionguard.com)os.environ["OPENAI_API_KEY"] = "<your OpenAI api key>"# Your Prediction Guard API key. Get one at predictionguard.comos.environ["PREDICTIONGUARD_TOKEN"] = "<your Prediction Guard access token>"
```

```
pgllm = PredictionGuard(model="OpenAI-text-davinci-003")
```

## Control the output structure/ type of LLMs[​](#control-the-output-structure-type-of-llms "Direct link to Control the output structure/ type of LLMs")

```
template = """Respond to the following query based on the context.Context: EVERY comment, DM + email suggestion has led us to this EXCITING announcement! 🎉 We have officially added TWO new candle subscription box options! 📦Exclusive Candle Box - $80 Monthly Candle Box - $45 (NEW!)Scent of The Month Box - $28 (NEW!)Head to stories to get ALLL the deets on each box! 👆 BONUS: Save 50% on your first box with code 50OFF! 🎉Query: {query}Result: """prompt = PromptTemplate.from_template(template)
```

```
# Without "guarding" or controlling the output of the LLM.pgllm(prompt.format(query="What kind of post is this?"))
```

```
# With "guarding" or controlling the output of the LLM. See the# Prediction Guard docs (https://docs.predictionguard.com) to learn how to# control the output with integer, float, boolean, JSON, and other types and# structures.pgllm = PredictionGuard(    model="OpenAI-text-davinci-003",    output={        "type": "categorical",        "categories": ["product announcement", "apology", "relational"],    },)pgllm(prompt.format(query="What kind of post is this?"))
```

## Chaining[​](#chaining "Direct link to Chaining")

```
pgllm = PredictionGuard(model="OpenAI-text-davinci-003")
```

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)llm_chain = LLMChain(prompt=prompt, llm=pgllm, verbose=True)question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.predict(question=question)
```

```
template = """Write a {adjective} poem about {subject}."""prompt = PromptTemplate.from_template(template)llm_chain = LLMChain(prompt=prompt, llm=pgllm, verbose=True)llm_chain.predict(adjective="sad", subject="ducks")
```

* * *

#### Help us out by providing feedback on this documentation page:
## Konko

> [Konko](https://www.konko.ai/) API is a fully managed Web API designed to help application developers:

1.  **Select** the right open source or proprietary LLMs for their application
2.  **Build** applications faster with integrations to leading application frameworks and fully managed APIs
3.  **Fine tune** smaller open-source LLMs to achieve industry-leading performance at a fraction of the cost
4.  **Deploy production-scale APIs** that meet security, privacy, throughput, and latency SLAs without infrastructure set-up or administration using Konko AI’s SOC 2 compliant, multi-cloud infrastructure

This example goes over how to use LangChain to interact with `Konko` completion [models](https://docs.konko.ai/docs/list-of-models#konko-hosted-models-for-completion)

To run this notebook, you’ll need Konko API key. Sign in to our web app to [create an API key](https://platform.konko.ai/settings/api-keys) to access models

#### Set Environment Variables[​](#set-environment-variables "Direct link to Set Environment Variables")

1.  You can set environment variables for
    1.  KONKO\_API\_KEY (Required)
    2.  OPENAI\_API\_KEY (Optional)
2.  In your current shell session, use the export command:

```
export KONKO_API_KEY={your_KONKO_API_KEY_here}export OPENAI_API_KEY={your_OPENAI_API_KEY_here} #Optional
```

## Calling a model[​](#calling-a-model "Direct link to Calling a model")

Find a model on the [Konko overview page](https://docs.konko.ai/docs/list-of-models)

Another way to find the list of models running on the Konko instance is through this [endpoint](https://docs.konko.ai/reference/get-models).

From here, we can initialize our model:

```
from langchain.llms import Konkollm = Konko(model="mistralai/mistral-7b-v0.1", temperature=0.1, max_tokens=128)input_ = """You are a helpful assistant. Explain Big Bang Theory briefly."""print(llm(input_))
```

```
Answer:The Big Bang Theory is a theory that explains the origin of the universe. According to the theory, the universe began with a single point of infinite density and temperature. This point is called the singularity. The singularity exploded and expanded rapidly. The expansion of the universe is still continuing.The Big Bang Theory is a theory that explains the origin of the universe. According to the theory, the universe began with a single point of infinite density and temperature. This point is called the singularity. The singularity exploded and expanded rapidly. The expansion of the universe is still continuing.Question
```
## RELLM

[RELLM](https://github.com/r2d4/rellm) is a library that wraps local Hugging Face pipeline models for structured decoding.

It works by generating tokens one at a time. At each step, it masks tokens that don’t conform to the provided partial regular expression.

**Warning - this module is still experimental**

```
%pip install --upgrade --quiet  rellm > /dev/null
```

### Hugging Face Baseline[​](#hugging-face-baseline "Direct link to Hugging Face Baseline")

First, let’s establish a qualitative baseline by checking the output of the model without structured decoding.

```
import logginglogging.basicConfig(level=logging.ERROR)prompt = """Human: "What's the capital of the United States?"AI Assistant:{  "action": "Final Answer",  "action_input": "The capital of the United States is Washington D.C."}Human: "What's the capital of Pennsylvania?"AI Assistant:{  "action": "Final Answer",  "action_input": "The capital of Pennsylvania is Harrisburg."}Human: "What 2 + 5?"AI Assistant:{  "action": "Final Answer",  "action_input": "2 + 5 = 7."}Human: 'What's the capital of Maryland?'AI Assistant:"""
```

```
from langchain_community.llms import HuggingFacePipelinefrom transformers import pipelinehf_model = pipeline(    "text-generation", model="cerebras/Cerebras-GPT-590M", max_new_tokens=200)original_model = HuggingFacePipeline(pipeline=hf_model)generated = original_model.generate([prompt], stop=["Human:"])print(generated)
```

```
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
```

```
generations=[[Generation(text=' "What\'s the capital of Maryland?"\n', generation_info=None)]] llm_output=None
```

**_That’s not so impressive, is it? It didn’t answer the question and it didn’t follow the JSON format at all! Let’s try with the structured decoder._**

## RELLM LLM Wrapper[​](#rellm-llm-wrapper "Direct link to RELLM LLM Wrapper")

Let’s try that again, now providing a regex to match the JSON structured format.

```
import regex  # Note this is the regex library NOT python's re stdlib module# We'll choose a regex that matches to a structured json string that looks like:# {#  "action": "Final Answer",# "action_input": string or dict# }pattern = regex.compile(    r'\{\s*"action":\s*"Final Answer",\s*"action_input":\s*(\{.*\}|"[^"]*")\s*\}\nHuman:')
```

```
from langchain_experimental.llms import RELLMmodel = RELLM(pipeline=hf_model, regex=pattern, max_new_tokens=200)generated = model.predict(prompt, stop=["Human:"])print(generated)
```

```
{"action": "Final Answer",  "action_input": "The capital of Maryland is Baltimore."}
```

**Voila! Free of parsing errors.**
_This community integration is deprecated. You should use [`ChatUpstage`](https://python.langchain.com/docs/integrations/chat/upstage/) instead to access Solar LLM via the chat model connector._

```
from langchain.chains import LLMChainfrom langchain_community.llms.solar import Solarfrom langchain_core.prompts import PromptTemplatetemplate = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)llm = Solar()llm_chain = LLMChain(prompt=prompt, llm=llm)question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)
```
Llamafile does this by combining [llama.cpp](https://github.com/ggerganov/llama.cpp) with [Cosmopolitan Libc](https://github.com/jart/cosmopolitan) into one framework that collapses all the complexity of LLMs down to a single-file executable (called a “llamafile”) that runs locally on most computers, with no installation.

Now you can make calls to the llamafile’s REST API. By default, the llamafile server listens at http://localhost:8080. You can find full server documentation [here](https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/README.md#api-endpoints). You can interact with the llamafile directly via the REST API, but here we’ll show how to interact with it using LangChain.

```
'? \nI\'ve got a thing for pink, but you know that.\n"Can we not talk about work anymore?" - What did she say?\nI don\'t want to be a burden on you.\nIt\'s hard to keep a good thing going.\nYou can\'t tell me what I want, I have a life too!'
```

```
.- She said, "I’m tired of my life. What should I do?"- The man replied, "I hear you. But don’t worry. Life is just like a joke. It has its funny parts too."- The woman looked at him, amazed and happy to hear his wise words. - "Thank you for your wisdom," she said, smiling. - He replied, "Any time. But it doesn't come easy. You have to laugh and keep moving forward in life."- She nodded, thanking him again. - The man smiled wryly. "Life can be tough. Sometimes it seems like you’re never going to get out of your situation."- He said, "I know that. But the key is not giving up. Life has many ups and downs, but in the end, it will turn out okay."- The woman's eyes softened. "Thank you for your advice. It's so important to keep moving forward in life," she said. - He nodded once again. "You’re welcome. I hope your journey is filled with laughter and joy."- They both smiled and left the bar, ready to embark on their respective adventures.
```

To learn more about the LangChain Expressive Language and the available methods on an LLM, see the [LCEL Interface](https://python.langchain.com/docs/expression_language/interface/)
## SparkLLM

[SparkLLM](https://xinghuo.xfyun.cn/spark) is a large-scale cognitive model independently developed by iFLYTEK. It has cross-domain knowledge and language understanding ability by learning a large amount of texts, codes and images. It can understand and perform tasks based on natural dialogue.

## Prerequisite[​](#prerequisite "Direct link to Prerequisite")

*   Get SparkLLM’s app\_id, api\_key and api\_secret from [iFlyTek SparkLLM API Console](https://console.xfyun.cn/services/bm3) (for more info, see [iFlyTek SparkLLM Intro](https://xinghuo.xfyun.cn/sparkapi) ), then set environment variables `IFLYTEK_SPARK_APP_ID`, `IFLYTEK_SPARK_API_KEY` and `IFLYTEK_SPARK_API_SECRET` or pass parameters when creating `ChatSparkLLM` as the demo above.

## Use SparkLLM[​](#use-sparkllm "Direct link to Use SparkLLM")

```
import osos.environ["IFLYTEK_SPARK_APP_ID"] = "app_id"os.environ["IFLYTEK_SPARK_API_KEY"] = "api_key"os.environ["IFLYTEK_SPARK_API_SECRET"] = "api_secret"
```

```
from langchain_community.llms import SparkLLM# Load the modelllm = SparkLLM()res = llm("What's your name?")print(res)
```

```
/Users/liugddx/code/langchain/libs/core/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.7 and will be removed in 0.2.0. Use invoke instead.  warn_deprecated(
```

```
My name is iFLYTEK Spark. How can I assist you today?
```

```
res = llm.generate(prompts=["hello!"])res
```

```
LLMResult(generations=[[Generation(text='Hello! How can I assist you today?')]], llm_output=None, run=[RunInfo(run_id=UUID('d8cdcd41-a698-4cbf-a28d-e74f9cd2037b'))])
```

```
for res in llm.stream("foo:"):    print(res)
```

```
Hello! How can I assist you today?
```

* * *

#### Help us out by providing feedback on this documentation page:
## Modal

The [Modal cloud platform](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. Use `modal` to run your own custom LLM models instead of depending on LLM APIs.

This example goes over how to use LangChain to interact with a `modal` HTTPS [web endpoint](https://modal.com/docs/guide/webhooks).

[_Question-answering with LangChain_](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example of how to use LangChain alonside `Modal`. In that example, Modal runs the LangChain application end-to-end and uses OpenAI as its LLM API.

```
%pip install --upgrade --quiet  modal
```

```
# Register an account with Modal and get a new token.!modal token new
```

```
Launching login page in your browser window...If this is not showing up, please copy this URL into your web browser manually:https://modal.com/token-flow/tf-Dzm3Y01234mqmm1234Vcu3
```

The [`langchain.llms.modal.Modal`](https://github.com/langchain-ai/langchain/blame/master/langchain/llms/modal.py) integration class requires that you deploy a Modal application with a web endpoint that complies with the following JSON interface:

1.  The LLM prompt is accepted as a `str` value under the key `"prompt"`
2.  The LLM response returned as a `str` value under the key `"prompt"`

**Example request JSON:**

```
{    "prompt": "Identify yourself, bot!",    "extra": "args are allowed",}
```

**Example response JSON:**

```
{    "prompt": "This is the LLM speaking",}
```

An example ‘dummy’ Modal web endpoint function fulfilling this interface would be

```
......class Request(BaseModel):    prompt: str@stub.function()@modal.web_endpoint(method="POST")def web(request: Request):    _ = request  # ignore input    return {"prompt": "hello world"}
```

*   See Modal’s [web endpoints](https://modal.com/docs/guide/webhooks#passing-arguments-to-web-endpoints) guide for the basics of setting up an endpoint that fulfils this interface.
*   See Modal’s [‘Run Falcon-40B with AutoGPTQ’](https://modal.com/docs/guide/ex/falcon_gptq) open-source LLM example as a starting point for your custom LLM!

Once you have a deployed Modal web endpoint, you can pass its URL into the `langchain.llms.modal.Modal` LLM class. This class can then function as a building block in your chain.

```
from langchain.chains import LLMChainfrom langchain_community.llms import Modalfrom langchain_core.prompts import PromptTemplate
```

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

```
endpoint_url = "https://ecorp--custom-llm-endpoint.modal.run"  # REPLACE ME with your deployed Modal web endpoint's URLllm = Modal(endpoint_url=endpoint_url)
```

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

```
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)
```
## TextGen

[GitHub:oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui) A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA.

This example goes over how to use LangChain to interact with LLM models via the `text-generation-webui` API integration.

Please ensure that you have `text-generation-webui` configured and an LLM installed. Recommended installation via the [one-click installer appropriate](https://github.com/oobabooga/text-generation-webui#one-click-installers) for your OS.

Once `text-generation-webui` is installed and confirmed working via the web interface, please enable the `api` option either through the web model configuration tab, or by adding the run-time arg `--api` to your start command.

## Set model\_url and run the example[​](#set-model_url-and-run-the-example "Direct link to Set model_url and run the example")

```
model_url = "http://localhost:5000"
```

```
from langchain.chains import LLMChainfrom langchain.globals import set_debugfrom langchain_community.llms import TextGenfrom langchain_core.prompts import PromptTemplateset_debug(True)template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)llm = TextGen(model_url=model_url)llm_chain = LLMChain(prompt=prompt, llm=llm)question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"llm_chain.run(question)
```

### Streaming Version[​](#streaming-version "Direct link to Streaming Version")

You should install websocket-client to use this feature. `pip install websocket-client`

```
model_url = "ws://localhost:5005"
```

```
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandlerfrom langchain.chains import LLMChainfrom langchain.globals import set_debugfrom langchain_community.llms import TextGenfrom langchain_core.prompts import PromptTemplateset_debug(True)template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)llm = TextGen(    model_url=model_url, streaming=True, callbacks=[StreamingStdOutCallbackHandler()])llm_chain = LLMChain(prompt=prompt, llm=llm)question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"llm_chain.run(question)
```

```
llm = TextGen(model_url=model_url, streaming=True)for chunk in llm.stream("Ask 'Hi, how are you?' like a pirate:'", stop=["'", "\n"]):    print(chunk, end="", flush=True)
```
## Minimax

[Minimax](https://api.minimax.chat/) is a Chinese startup that provides natural language processing models for companies and individuals.

This example demonstrates using Langchain to interact with Minimax.

## Setup

To run this notebook, you’ll need a [Minimax account](https://api.minimax.chat/), an [API key](https://api.minimax.chat/user-center/basic-information/interface-key), and a [Group ID](https://api.minimax.chat/user-center/basic-information)

## Single model call

```
from langchain_community.llms import Minimax
```

```
# Load the modelminimax = Minimax(minimax_api_key="YOUR_API_KEY", minimax_group_id="YOUR_GROUP_ID")
```

```
# Prompt the modelminimax("What is the difference between panda and bear?")
```

## Chained model calls

```
# get api_key and group_id: https://api.minimax.chat/user-center/basic-information# We need `MINIMAX_API_KEY` and `MINIMAX_GROUP_ID`import osos.environ["MINIMAX_API_KEY"] = "YOUR_API_KEY"os.environ["MINIMAX_GROUP_ID"] = "YOUR_GROUP_ID"
```

```
from langchain.chains import LLMChainfrom langchain_community.llms import Minimaxfrom langchain_core.prompts import PromptTemplate
```

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

```
question = "What NBA team won the Championship in the year Jay Zhou was born?"llm_chain.run(question)
```

* * *

#### Help us out by providing feedback on this documentation page:
## MoonshotChat

[Moonshot](https://platform.moonshot.cn/) is a Chinese startup that provides LLM service for companies and individuals.

This example goes over how to use LangChain to interact with Moonshot.

```
from langchain_community.llms.moonshot import Moonshot
```

```
import os# Generate your api key from: https://platform.moonshot.cn/console/api-keysos.environ["MOONSHOT_API_KEY"] = "MOONSHOT_API_KEY"
```

```
llm = Moonshot()# or use a specific model# Available models: https://platform.moonshot.cn/docs# llm = Moonshot(model="moonshot-v1-128k")
```

```
# Prompt the modelllm.invoke("What is the difference between panda and bear?")
```

* * *

#### Help us out by providing feedback on this documentation page:
## Together AI

> The Together API makes it easy to fine-tune or run leading open-source models with a couple lines of code. We have integrated the world’s leading open-source models, including Llama-2, RedPajama, Falcon, Alpaca, Stable Diffusion XL, and more. Read more: [https://together.ai](https://together.ai/)

To use, you’ll need an API key which you can find here: [https://api.together.xyz/settings/api-keys](https://api.together.xyz/settings/api-keys). This can be passed in as init param `together_api_key` or set as environment variable `TOGETHER_API_KEY`.

Together API reference: [https://docs.together.ai/reference](https://docs.together.ai/reference)

```
%pip install --upgrade --quiet langchain-together
```

```
from langchain_together import Togetherllm = Together(    model="togethercomputer/RedPajama-INCITE-7B-Base",    temperature=0.7,    max_tokens=128,    top_k=1,    # together_api_key="...")input_ = """You are a teacher with a deep knowledge of machine learning and AI. \You provide succinct and accurate answers. Answer the following question: What is a large language model?"""print(llm.invoke(input_))
```

```
A: A large language model is a neural network that is trained on a large amount of text data. It is able to generate text that is similar to the training data, and can be used for tasks such as language translation, question answering, and text summarization.A: A large language model is a neural network that is trained on a large amount of text data. It is able to generate text that is similar to the training data, and can be used for tasks such as language translation, question answering, and text summarization.A: A large language model is a neural network that is trained on
```

* * *

#### Help us out by providing feedback on this documentation page:
[Nebula](https://symbl.ai/nebula/) is a large language model (LLM) built by [Symbl.ai](https://symbl.ai/). It is trained to perform generative tasks on human conversations. Nebula excels at modeling the nuanced details of a conversation and performing tasks on the conversation.

This example goes over how to use LangChain to interact with the [Nebula platform](https://docs.symbl.ai/docs/nebula-llm).

```
from langchain_community.llms.symblai_nebula import Nebulallm = Nebula(nebula_api_key="<your_api_key>")
```

Use a conversation transcript and instruction to construct a prompt.

```
from langchain.chains import LLMChainfrom langchain_core.prompts import PromptTemplateconversation = """Sam: Good morning, team! Let's keep this standup concise. We'll go in the usual order: what you did yesterday, what you plan to do today, and any blockers. Alex, kick us off.Alex: Morning! Yesterday, I wrapped up the UI for the user dashboard. The new charts and widgets are now responsive. I also had a sync with the design team to ensure the final touchups are in line with the brand guidelines. Today, I'll start integrating the frontend with the new API endpoints Rhea was working on. The only blocker is waiting for some final API documentation, but I guess Rhea can update on that.Rhea: Hey, all! Yep, about the API documentation - I completed the majority of the backend work for user data retrieval yesterday. The endpoints are mostly set up, but I need to do a bit more testing today. I'll finalize the API documentation by noon, so that should unblock Alex. After that, I’ll be working on optimizing the database queries for faster data fetching. No other blockers on my end.Sam: Great, thanks Rhea. Do reach out if you need any testing assistance or if there are any hitches with the database. Now, my update: Yesterday, I coordinated with the client to get clarity on some feature requirements. Today, I'll be updating our project roadmap and timelines based on their feedback. Additionally, I'll be sitting with the QA team in the afternoon for preliminary testing. Blocker: I might need both of you to be available for a quick call in case the client wants to discuss the changes live.Alex: Sounds good, Sam. Just let us know a little in advance for the call.Rhea: Agreed. We can make time for that.Sam: Perfect! Let's keep the momentum going. Reach out if there are any sudden issues or support needed. Have a productive day!Alex: You too.Rhea: Thanks, bye!"""instruction = "Identify the main objectives mentioned in this conversation."prompt = PromptTemplate.from_template("{instruction}\n{conversation}")llm_chain = LLMChain(prompt=prompt, llm=llm)llm_chain.run(instruction=instruction, conversation=conversation)
```
## MLX Local Pipelines

MLX models can be run locally through the `MLXPipeline` class.

The [MLX Community](https://huggingface.co/mlx-community) hosts over 150 models, all open source and publicly available on Hugging Face Model Hub a online platform where people can easily collaborate and build ML together.

These can be called from LangChain either through this local pipeline wrapper or by calling their hosted inference endpoints through the MlXPipeline class. For more information on mlx, see the [examples repo](https://github.com/ml-explore/mlx-examples/tree/main/llms) notebook.

To use, you should have the `mlx-lm` python [package installed](https://pypi.org/project/mlx-lm/), as well as [transformers](https://pypi.org/project/transformers/). You can also install `huggingface_hub`.

```
%pip install --upgrade --quiet  mlx-lm transformers huggingface_hub
```

### Model Loading[​](#model-loading "Direct link to Model Loading")

Models can be loaded by specifying the model parameters using the `from_model_id` method.

```
from langchain_community.llms.mlx_pipeline import MLXPipelinepipe = MLXPipeline.from_model_id(    "mlx-community/quantized-gemma-2b-it",    pipeline_kwargs={"max_tokens": 10, "temp": 0.1},)
```

They can also be loaded by passing in an existing `transformers` pipeline directly

```
from langchain_community.llms.huggingface_pipeline import HuggingFacePipelinefrom mlx_lm import loadmodel, tokenizer = load("mlx-community/quantized-gemma-2b-it")pipe = MLXPipeline(model=model, tokenizer=tokenizer)
```

### Create Chain[​](#create-chain "Direct link to Create Chain")

With the model loaded into memory, you can compose it with a prompt to form a chain.

```
from langchain_core.prompts import PromptTemplatetemplate = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)chain = prompt | pipequestion = "What is electroencephalography?"print(chain.invoke({"question": question}))
```
## StochasticAI

> [Stochastic Acceleration Platform](https://docs.stochastic.ai/docs/introduction/) aims to simplify the life cycle of a Deep Learning model. From uploading and versioning the model, through training, compression and acceleration to putting it into production.

This example goes over how to use LangChain to interact with `StochasticAI` models.

You have to get the API\_KEY and the API\_URL [here](https://app.stochastic.ai/workspace/profile/settings?tab=profile).

```
from getpass import getpassSTOCHASTICAI_API_KEY = getpass()
```

```
import osos.environ["STOCHASTICAI_API_KEY"] = STOCHASTICAI_API_KEY
```

```
from langchain.chains import LLMChainfrom langchain_community.llms import StochasticAIfrom langchain_core.prompts import PromptTemplate
```

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

```
llm = StochasticAI(api_url=YOUR_API_URL)
```

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

```
question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"llm_chain.run(question)
```

```
"\n\nStep 1: In 1999, the St. Louis Rams won the Super Bowl.\n\nStep 2: In 1999, Beiber was born.\n\nStep 3: The Rams were in Los Angeles at the time.\n\nStep 4: So they didn't play in the Super Bowl that year.\n"
```

* * *

#### Help us out by providing feedback on this documentation page:
## MosaicML

[MosaicML](https://docs.mosaicml.com/en/latest/inference.html) offers a managed inference service. You can either use a variety of open-source models, or deploy your own.

This example goes over how to use LangChain to interact with MosaicML Inference for text completion.

```
# sign up for an account: https://forms.mosaicml.com/demo?utm_source=langchainfrom getpass import getpassMOSAICML_API_TOKEN = getpass()
```

```
import osos.environ["MOSAICML_API_TOKEN"] = MOSAICML_API_TOKEN
```

```
from langchain.chains import LLMChainfrom langchain_community.llms import MosaicMLfrom langchain_core.prompts import PromptTemplate
```

```
template = """Question: {question}"""prompt = PromptTemplate.from_template(template)
```

```
llm = MosaicML(inject_instruction_format=True, model_kwargs={"max_new_tokens": 128})
```

```
llm_chain = LLMChain(prompt=prompt, llm=llm)
```

```
question = "What is one good reason why you should train a large language model on domain specific data?"llm_chain.run(question)
```

* * *

#### Help us out by providing feedback on this documentation page:
## Tongyi Qwen

Tongyi Qwen is a large-scale language model developed by Alibaba’s Damo Academy. It is capable of understanding user intent through natural language understanding and semantic analysis, based on user input in natural language. It provides services and assistance to users in different domains and tasks. By providing clear and detailed instructions, you can obtain results that better align with your expectations.

## Setting up[​](#setting-up "Direct link to Setting up")

```
# Install the package%pip install --upgrade --quiet  dashscope
```

```
# Get a new token: https://help.aliyun.com/document_detail/611472.html?spm=a2c4g.2399481.0.0from getpass import getpassDASHSCOPE_API_KEY = getpass()
```

```
import osos.environ["DASHSCOPE_API_KEY"] = DASHSCOPE_API_KEY
```

```
from langchain_community.llms import Tongyi
```

```
Tongyi().invoke("What NFL team won the Super Bowl in the year Justin Bieber was born?")
```

```
'Justin Bieber was born on March 1, 1994. The Super Bowl that took place in the same year was Super Bowl XXVIII, which was played on January 30, 1994. The winner of that Super Bowl was the Dallas Cowboys, who defeated the Buffalo Bills with a score of 30-13.'
```

## Using in a chain[​](#using-in-a-chain "Direct link to Using in a chain")

```
from langchain_core.prompts import PromptTemplate
```

```
template = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)
```

```
question = "What NFL team won the Super Bowl in the year Justin Bieber was born?"chain.invoke({"question": question})
```

```
'Justin Bieber was born on March 1, 1994. The Super Bowl that took place in the same calendar year was Super Bowl XXVIII, which was played on January 30, 1994. The winner of Super Bowl XXVIII was the Dallas Cowboys, who defeated the Buffalo Bills with a score of 30-13.'
```

* * *

#### Help us out by providing feedback on this documentation page: