## MLflow Deployments for LLMs

> [The MLflow Deployments for LLMs](https://www.mlflow.org/docs/latest/llms/deployments/index.html) is a powerful tool designed to streamline the usage and management of various large language model (LLM) providers, such as OpenAI and Anthropic, within an organization. It offers a high-level interface that simplifies the interaction with these services by providing a unified endpoint to handle specific LLM related requests.

## Installation and Setup[​](#installation-and-setup "Direct link to Installation and Setup")

Install `mlflow` with MLflow Deployments dependencies:

```
pip install 'mlflow[genai]'
```

Set the OpenAI API key as an environment variable:

```
export OPENAI_API_KEY=...
```

Create a configuration file:

```
endpoints:  - name: completions    endpoint_type: llm/v1/completions    model:      provider: openai      name: text-davinci-003      config:        openai_api_key: $OPENAI_API_KEY  - name: embeddings    endpoint_type: llm/v1/embeddings    model:      provider: openai      name: text-embedding-ada-002      config:        openai_api_key: $OPENAI_API_KEY
```

Start the deployments server:

```
mlflow deployments start-server --config-path /path/to/config.yaml
```

## Example provided by `MLflow`[​](#example-provided-by-mlflow "Direct link to example-provided-by-mlflow")

> The `mlflow.langchain` module provides an API for logging and loading `LangChain` models. This module exports multivariate LangChain models in the langchain flavor and univariate LangChain models in the pyfunc flavor.

See the [API documentation and examples](https://www.mlflow.org/docs/latest/llms/langchain/index.html) for more information.

## Completions Example[​](#completions-example "Direct link to Completions Example")

```
import mlflowfrom langchain.chains import LLMChain, PromptTemplatefrom langchain_community.llms import Mlflowllm = Mlflow(    target_uri="http://127.0.0.1:5000",    endpoint="completions",)llm_chain = LLMChain(    llm=Mlflow,    prompt=PromptTemplate(        input_variables=["adjective"],        template="Tell me a {adjective} joke",    ),)result = llm_chain.run(adjective="funny")print(result)with mlflow.start_run():    model_info = mlflow.langchain.log_model(chain, "model")model = mlflow.pyfunc.load_model(model_info.model_uri)print(model.predict([{"adjective": "funny"}]))
```

## Embeddings Example[​](#embeddings-example "Direct link to Embeddings Example")

```
from langchain_community.embeddings import MlflowEmbeddingsembeddings = MlflowEmbeddings(    target_uri="http://127.0.0.1:5000",    endpoint="embeddings",)print(embeddings.embed_query("hello"))print(embeddings.embed_documents(["hello"]))
```

## Chat Example[​](#chat-example "Direct link to Chat Example")

```
from langchain_community.chat_models import ChatMlflowfrom langchain_core.messages import HumanMessage, SystemMessagechat = ChatMlflow(    target_uri="http://127.0.0.1:5000",    endpoint="chat",)messages = [    SystemMessage(        content="You are a helpful assistant that translates English to French."    ),    HumanMessage(        content="Translate this sentence from English to French: I love programming."    ),]print(chat(messages))
```