Integrate Orquesta with OpenAI using Python SDK
Tags: openai, llm, promptengineering, ai
Published: 2023-09-18 13:09:25

[Orquesta](https://orquesta.cloud/) is a powerful LLM Ops suite designed to manage both public and private LLMs (Large Language Models) from a single source. It offers full transparency on performance and costs while reducing your release cycles from weeks to minutes. Integrating Orquesta into your current setup or a new workflow requires a couple of lines of code, ensuring seamless collaboration and transparency in prompt engineering and prompt management for your team.

With Orquesta, you gain access to several LLM Ops features, enabling your team to:

- Collaborate directly across product, engineering, and domain expert teams.

- Manage prompts for both public and private LLM models.

- Customize and localize prompt variants based on your data model.

- Push new versions directly to production and roll back instantly.

- Obtain model-specific token and cost estimates.

- Gain insights into model-specific costs, performance, and latency.

- Gather both quantitative and qualitative end-user feedback.

- Experiment in production and gather real-world feedback.

- Make decisions grounded in real-world information.

This article guides you through integrating your SaaS with Orquesta and OpenAI using our Python SDK. By the end of the article, you'll know how to set up a prompt in Orquesta, perform prompt engineering, request a prompt variant using the SDK code generator, map the Orquesta response with OpenAI, send a payload to OpenAI, and report the response back to Orquesta for observability and monitoring.

## Prerequisites
For you to be able to follow along in this tutorial, you will need the following:

- Jupyter Notebook (or any IDE of your choice).

- An OpenAI account, you can sign up [here](https://platform.openai.com/signup?launch).

- Orquesta Python SDK.


## Integration

Follow these steps to integrate the Python SDK with OpenAI.

### Step 1 - Install SDK and create a client instance

```python
pip install orquesta-sdk
```

To create a client instance, you need to have access to the Orquesta API key, which can be found in your workspace `https://my.orquesta.dev/<workspace-name>/settings/developers`.

![api key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zhedhfcadcsi88ow5jd9.png)

Copy it and add the following code to your notebook to initialize the Orquesta client.

```python
from orquesta_sdk import OrquestaClient, OrquestaClientOptions

api_key = "<ORQUESTA_API_KEY>"

options = OrquestaClientOptions(
    api_key=api_key,
    ttl=3600
)

client = OrquestaClient(options)
```

The `OrquestaClient` and the `OrquestaClientOptions` classes which are already defined in the `orquesta_sdk` module is imported. The **API key**, which is used for authentication, is assigned to the variable `api_key`, you can either add the API key this way, or you can add it using the environment variable; `api_key = os.environ.get("ORQUESTA_API_KEY", "__API_KEY__")`. The instance of the `OrquestaClientOptions` class is created and configured with the `api_key` and the `ttl` (Time to Live) in seconds for the local cache; by default, it is **3600** seconds (1 hour).

Finally, an instance of the `OrquestaClient` class is created and initialized with the previously configured options object. This `client` instance can now interact with the Orquesta service using the provided API key for authentication.

### Step 2 - Set up a prompt and its variants

After successfully connecting to Orquesta, you continue within the Orquesta Admin Panel to set up your prompt and variants. A prompt is the specific task you provide to LLM, and you'll get a response that is the output of the language model accomplishing the task. To create a prompt, click on **Add Prompt** and the **prompt key**.

![Set up a prompt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xmbgwo2k2bq1zgeapnow.png)


![Set up its variants](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6qzve428enp5ewwgr9yy.png)

The image above represents the Prompt Studio in Orquesta, where:

1. The name of the prompt variant. 

2. Notes, this is where you drop notes for other collaborators.

3. Since we are working on a chat prompt, this is where you manage the System-User-Assistant messages.

4. Prompt variables provide flexibility in your prompts.

5. Prompt tokens and cost are estimated based on the model selected.

6. Model Selector.

7. Click **Save** once you are done.

### Step 3 - Request a variant from Orquesta using the SDK 

Our flexible configuration matrix allows you to define multiple prompt variants based on custom context. This allows you to work with different prompts and hyperparameters with, for example, environment, country, locale or user segment. The Code Snippet Generator makes it easy to request a prompt variant. 


![Request a variant from Orquesta](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/onn10mco2ol1jxjmwzvr.png)

Once you open the Code Snippet Generator, you can use the generated snippet to consume your first prompt from your application.


![Request a variant from Orquesta](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/az649ndelilop6h4dyjm.png)

### Step 4 - Map the Orquesta response to OpenAI using a Helper
Map the Orquesta response to OpenAI's API using the Helper functions. Each LLM provider has its own Helper function in Orquesta. 

For OpenAI, use the Helper: `orquesta_openai_parameters_mapper`or Class: `OrquestaOpenAIPromptParameters`.

```python
import os
import time
import openai
from orquesta_sdk import OrquestaClient, OrquestaClientOptions
from orquesta_sdk.helpers import orquesta_openai_parameters_mapper
from orquesta_sdk.prompts import OrquestaPromptMetrics

openai.api_key = "<OPENAI_API_KEY>"

```

Paste the code copied from the Code Snippet Generator here.

```python
# Query the prompt from Orquesta
prompt = client.prompts.query(
  key="customer-support-chat",
  context={
    "environments": ["test"],
    "country": ["BEL", "NLD"],
    "locale": ["en"],
    "user-segment": ["b2c"]
  },
  variables={ "customer_name": "John" },
  metadata={"user_id":45515}
)

if prompt.has_error:
    print("There was an error while fetching the prompt")
```

You can now send the payload to OpenAI and receive the response.

```python
# Start time of the completion request
start_time = time.time()
print(f'Start time: {start_time}')

completion = openai.ChatCompletion.create(
    **orquesta_openai_parameters_mapper(prompt.value),
    model=prompt.value.get("model"),
    messages=prompt.value.get("messages"),
)

# End time of the completion request
end_time = time.time()
print(f'End time: {end_time}')

# Calculate the difference (latency) in milliseconds
latency = (end_time - start_time) * 1000
print(f'Latency is: {latency}')
```


![Latency](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/reflmtihnt8hnaf426ys.png)

### Step 5 - Report analytics back to Orquesta

After each query, Orquesta generates a log with a Trace ID. Using the `add_metrics()` method, you can add additional information, such as the **llm_response**, **metadata**, **latency**, and **economics**. 

```python
# Report the metrics back to Orquesta
metrics = OrquestaPromptMetrics(
    economics={
        "total_tokens": completion.usage.get("total_tokens"),
        "completion_tokens": completion.usage.get("completion_tokens"),
        "prompt_tokens": completion.usage.get("prompt_tokens"),
    },
    llm_response=completion.choices[0].message.content,
    latency=latency,
    metadata={
        "finish_reason": completion.choices[0].finish_reason,
    },
)

prompt.add_metrics(metrics=metrics)
```

## Conclusion
And that is it, and you have integrated Orquesta with OpenAI using the Python SDK! You can easily design, test, and manage prompts for all your LLM providers using Orquesta by simply leveraging its power tools with real-time logs, versioning, code snippets, and a playground for your prompts. 

Orquesta supports other [SDKs](https://orquesta.cloud/developers/sdk-overview) such as Angular, Node.js, React, and TypeScript. Refer to our [documentation](https://orquesta.cloud/developers/api-introduction) for more information.

## Full Code Example

```python
import os
import time
import openai
from orquesta_sdk import OrquestaClient, OrquestaClientOptions
from orquesta_sdk.helpers import orquesta_openai_parameters_mapper
from orquesta_sdk.prompts import OrquestaPromptMetrics

openai.api_key = "<OPENAI_API_KEY>"

# Initialize Orquesta client

api_key = "<ORQUESTA_API_KEY>"
options = OrquestaClientOptions(
    api_key=api_key,
    ttl=3600
)

client = OrquestaClient(options)

# Query the prompt from Orquesta
prompt = client.prompts.query(
  key="customer-support-chat",
  context={
    "environments": ["test"],
    "country": ["BEL", "NLD"],
    "locale": ["en"],
    "user-segment": ["b2c"]
  },
  variables={
    "customer_name": "John"
  },
  metadata={"user_id":45515}
)

if prompt.has_error:
    print("There was an error while fetching the prompt")

# Start time of the completion request
start_time = time.time()
print(f'Start time: {start_time}')

completion = openai.ChatCompletion.create(
    **orquesta_openai_parameters_mapper(prompt.value),
    model=prompt.value.get("model"),
    messages=prompt.value.get("messages"),
)

# End time of the completion request
end_time = time.time()
print(f'End time: {end_time}')

# Calculate the difference (latency) in milliseconds
latency = (end_time - start_time) * 1000
print(f'Latency is: {latency}')

# Report the metrics back to Orquesta
metrics = OrquestaPromptMetrics(
    economics={
        "total_tokens": completion.usage.get("total_tokens"),
        "completion_tokens": completion.usage.get("completion_tokens"),
        "prompt_tokens": completion.usage.get("prompt_tokens"),
    },
    llm_response=completion.choices[0].message.content,
    latency=latency,
    metadata={
        "finish_reason": completion.choices[0].finish_reason,
    },
)

prompt.add_metrics(metrics=metrics)
```