seansullivan/Langchain-Docs huggingface dataset:
You can make your own custom string evaluators by inheriting from the `StringEvaluator` class and implementing the `_evaluate_strings` (and `_aevaluate_strings` for async support) methods.

In this example, you will create a perplexity evaluator using the HuggingFace [evaluate](https://huggingface.co/docs/evaluate/index) library. [Perplexity](https://en.wikipedia.org/wiki/Perplexity) is a measure of how well the generated text would be predicted by the model used to compute the metric.

```
from typing import Any, Optionalfrom evaluate import loadfrom langchain.evaluation import StringEvaluatorclass PerplexityEvaluator(StringEvaluator):    """Evaluate the perplexity of a predicted string."""    def __init__(self, model_id: str = "gpt2"):        self.model_id = model_id        self.metric_fn = load(            "perplexity", module_type="metric", model_id=self.model_id, pad_token=0        )    def _evaluate_strings(        self,        *,        prediction: str,        reference: Optional[str] = None,        input: Optional[str] = None,        **kwargs: Any,    ) -> dict:        results = self.metric_fn.compute(            predictions=[prediction], model_id=self.model_id        )        ppl = results["perplexities"][0]        return {"score": ppl}
```

```
Using pad_token, but it is not set yet.
```

```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...To disable this warning, you can either:    - Avoid using `tokenizers` before the fork if possible    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
```

```
  0%|          | 0/1 [00:00<?, ?it/s]
```

```
{'score': 190.3675537109375}
```

```
Using pad_token, but it is not set yet.
```

```
  0%|          | 0/1 [00:00<?, ?it/s]
```

```
{'score': 1982.0709228515625}
```
## Exact Match

[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/exact_match.ipynb)

Open In Colab

Probably the simplest ways to evaluate an LLM or runnable’s string output against a reference label is by a simple string equivalence.

This can be accessed using the `exact_match` evaluator.

```
from langchain.evaluation import ExactMatchStringEvaluatorevaluator = ExactMatchStringEvaluator()
```

Alternatively via the loader:

```
from langchain.evaluation import load_evaluatorevaluator = load_evaluator("exact_match")
```

```
evaluator.evaluate_strings(    prediction="1 LLM.",    reference="2 llm",)
```

```
evaluator.evaluate_strings(    prediction="LangChain",    reference="langchain",)
```

## Configure the ExactMatchStringEvaluator[​](#configure-the-exactmatchstringevaluator "Direct link to Configure the ExactMatchStringEvaluator")

You can relax the “exactness” when comparing strings.

```
evaluator = ExactMatchStringEvaluator(    ignore_case=True,    ignore_numbers=True,    ignore_punctuation=True,)# Alternatively# evaluator = load_evaluator("exact_match", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)
```

```
evaluator.evaluate_strings(    prediction="1 LLM.",    reference="2 llm",)
```
## Regex Match

[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/regex_match.ipynb)

Open In Colab

To evaluate chain or runnable string predictions against a custom regex, you can use the `regex_match` evaluator.

```
from langchain.evaluation import RegexMatchStringEvaluatorevaluator = RegexMatchStringEvaluator()
```

Alternatively via the loader:

```
from langchain.evaluation import load_evaluatorevaluator = load_evaluator("regex_match")
```

```
# Check for the presence of a YYYY-MM-DD string.evaluator.evaluate_strings(    prediction="The delivery will be made on 2024-01-05",    reference=".*\\b\\d{4}-\\d{2}-\\d{2}\\b.*",)
```

```
# Check for the presence of a MM-DD-YYYY string.evaluator.evaluate_strings(    prediction="The delivery will be made on 2024-01-05",    reference=".*\\b\\d{2}-\\d{2}-\\d{4}\\b.*",)
```

```
# Check for the presence of a MM-DD-YYYY string.evaluator.evaluate_strings(    prediction="The delivery will be made on 01-05-2024",    reference=".*\\b\\d{2}-\\d{2}-\\d{4}\\b.*",)
```

## Match against multiple patterns[​](#match-against-multiple-patterns "Direct link to Match against multiple patterns")

To match against multiple patterns, use a regex union “|”.

```
# Check for the presence of a MM-DD-YYYY string or YYYY-MM-DDevaluator.evaluate_strings(    prediction="The delivery will be made on 01-05-2024",    reference="|".join(        [".*\\b\\d{4}-\\d{2}-\\d{2}\\b.*", ".*\\b\\d{2}-\\d{2}-\\d{4}\\b.*"]    ),)
```

## Configure the RegexMatchStringEvaluator[​](#configure-the-regexmatchstringevaluator "Direct link to Configure the RegexMatchStringEvaluator")

You can specify any regex flags to use when matching.

```
import reevaluator = RegexMatchStringEvaluator(flags=re.IGNORECASE)# Alternatively# evaluator = load_evaluator("exact_match", flags=re.IGNORECASE)
```

```
evaluator.evaluate_strings(    prediction="I LOVE testing",    reference="I love testing",)
```
## String Distance

[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/evaluation/string/string_distance.ipynb)

Open In Colab

> In information theory, linguistics, and computer science, the [Levenshtein distance (Wikipedia)](https://en.wikipedia.org/wiki/Levenshtein_distance) is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other. It is named after the Soviet mathematician Vladimir Levenshtein, who considered this distance in 1965.

One of the simplest ways to compare an LLM or chain’s string output against a reference label is by using string distance measurements such as `Levenshtein` or `postfix` distance. This can be used alongside approximate/fuzzy matching criteria for very basic unit testing.

This can be accessed using the `string_distance` evaluator, which uses distance metrics from the [rapidfuzz](https://github.com/maxbachmann/RapidFuzz) library.

**Note:** The returned scores are _distances_, meaning lower is typically “better”.

For more information, check out the reference docs for the [StringDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.string_distance.base.StringDistanceEvalChain.html#langchain.evaluation.string_distance.base.StringDistanceEvalChain) for more info.

```
%pip install --upgrade --quiet  rapidfuzz
```

```
from langchain.evaluation import load_evaluatorevaluator = load_evaluator("string_distance")
```

```
evaluator.evaluate_strings(    prediction="The job is completely done.",    reference="The job is done",)
```

```
{'score': 0.11555555555555552}
```

```
# The results purely character-based, so it's less useful when negation is concernedevaluator.evaluate_strings(    prediction="The job is done.",    reference="The job isn't done",)
```

```
{'score': 0.0724999999999999}
```

## Configure the String Distance Metric[​](#configure-the-string-distance-metric "Direct link to Configure the String Distance Metric")

By default, the `StringDistanceEvalChain` uses levenshtein distance, but it also supports other string distance algorithms. Configure using the `distance` argument.

```
from langchain.evaluation import StringDistancelist(StringDistance)
```

```
[<StringDistance.DAMERAU_LEVENSHTEIN: 'damerau_levenshtein'>, <StringDistance.LEVENSHTEIN: 'levenshtein'>, <StringDistance.JARO: 'jaro'>, <StringDistance.JARO_WINKLER: 'jaro_winkler'>]
```

```
jaro_evaluator = load_evaluator("string_distance", distance=StringDistance.JARO)
```

```
jaro_evaluator.evaluate_strings(    prediction="The job is completely done.",    reference="The job is done",)
```

```
{'score': 0.19259259259259254}
```

```
jaro_evaluator.evaluate_strings(    prediction="The job is done.",    reference="The job isn't done",)
```

```
{'score': 0.12083333333333324}
```

* * *

#### Help us out by providing feedback on this documentation page:
## Trajectory Evaluators

Trajectory Evaluators in LangChain provide a more holistic approach to evaluating an agent. These evaluators assess the full sequence of actions taken by an agent and their corresponding responses, which we refer to as the "trajectory". This allows you to better measure an agent's effectiveness and capabilities.

A Trajectory Evaluator implements the `AgentTrajectoryEvaluator` interface, which requires two main methods:

*   `evaluate_agent_trajectory`: This method synchronously evaluates an agent's trajectory.
*   `aevaluate_agent_trajectory`: This asynchronous counterpart allows evaluations to be run in parallel for efficiency.

Both methods accept three main parameters:

*   `input`: The initial input given to the agent.
*   `prediction`: The final predicted response from the agent.
*   `agent_trajectory`: The intermediate steps taken by the agent, given as a list of tuples.

These methods return a dictionary. It is recommended that custom implementations return a `score` (a float indicating the effectiveness of the agent) and `reasoning` (a string explaining the reasoning behind the score).

You can capture an agent's trajectory by initializing the agent with the `return_intermediate_steps=True` parameter. This lets you collect all intermediate steps without relying on special callbacks.

For a deeper dive into the implementation and use of Trajectory Evaluators, refer to the sections below.

[

## 📄️ Custom Trajectory Evaluator

Open In Colab

](https://python.langchain.com/docs/guides/productionization/evaluation/trajectory/custom/)

[

## 📄️ Agent Trajectory

Open In Colab

](https://python.langchain.com/docs/guides/productionization/evaluation/trajectory/trajectory_eval/)
You can make your own custom trajectory evaluators by inheriting from the [AgentTrajectoryEvaluator](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.schema.AgentTrajectoryEvaluator.html#langchain.evaluation.schema.AgentTrajectoryEvaluator) class and overwriting the `_evaluate_agent_trajectory` (and `_aevaluate_agent_action`) method.

In this example, you will make a simple trajectory evaluator that uses an LLM to determine if any actions were unnecessary.

```
from typing import Any, Optional, Sequence, Tuplefrom langchain.chains import LLMChainfrom langchain.evaluation import AgentTrajectoryEvaluatorfrom langchain_core.agents import AgentActionfrom langchain_openai import ChatOpenAIclass StepNecessityEvaluator(AgentTrajectoryEvaluator):    """Evaluate the perplexity of a predicted string."""    def __init__(self) -> None:        llm = ChatOpenAI(model="gpt-4", temperature=0.0)        template = """Are any of the following steps unnecessary in answering {input}? Provide the verdict on a new line as a single "Y" for yes or "N" for no.        DATA        ------        Steps: {trajectory}        ------        Verdict:"""        self.chain = LLMChain.from_string(llm, template)    def _evaluate_agent_trajectory(        self,        *,        prediction: str,        input: str,        agent_trajectory: Sequence[Tuple[AgentAction, str]],        reference: Optional[str] = None,        **kwargs: Any,    ) -> dict:        vals = [            f"{i}: Action=[{action.tool}] returned observation = [{observation}]"            for i, (action, observation) in enumerate(agent_trajectory)        ]        trajectory = "\n".join(vals)        response = self.chain.run(dict(trajectory=trajectory, input=input), **kwargs)        decision = response.split("\n")[-1].strip()        score = 1 if decision == "Y" else 0        return {"score": score, "value": decision, "reasoning": response}
```

The example above will return a score of 1 if the language model predicts that any of the actions were unnecessary, and it returns a score of 0 if all of them were predicted to be necessary. It returns the string ‘decision’ as the ‘value’, and includes the rest of the generated text as ‘reasoning’ to let you audit the decision.

You can call this evaluator to grade the intermediate steps of your agent’s trajectory.

```
evaluator = StepNecessityEvaluator()evaluator.evaluate_agent_trajectory(    prediction="The answer is pi",    input="What is today?",    agent_trajectory=[        (            AgentAction(tool="ask", tool_input="What is today?", log=""),            "tomorrow's yesterday",        ),        (            AgentAction(tool="check_tv", tool_input="Watch tv for half hour", log=""),            "bzzz",        ),    ],)
```

```
{'score': 1, 'value': 'Y', 'reasoning': 'Y'}
```
## Privacy & Safety

One of the key concerns with using LLMs is that they may misuse private data or generate harmful or unethical text. This is an area of active research in the field. Here we present some built-in chains inspired by this research, which are intended to make the outputs of LLMs safer.

*   [Amazon Comprehend moderation chain](https://python.langchain.com/docs/guides/productionization/safety/amazon_comprehend_chain/): Use [Amazon Comprehend](https://aws.amazon.com/comprehend/) to detect and handle Personally Identifiable Information (PII) and toxicity.
*   [Constitutional chain](https://python.langchain.com/docs/guides/productionization/safety/constitutional_chain/): Prompt the model with a set of principles which should guide the model behavior.
*   [Hugging Face prompt injection identification](https://python.langchain.com/docs/guides/productionization/safety/hugging_face_prompt_injection/): Detect and handle prompt injection attacks.
*   [Layerup Security](https://python.langchain.com/docs/guides/productionization/safety/layerup_security/): Easily mask PII & sensitive data, detect and mitigate 10+ LLM-based threat vectors, including PII & sensitive data, prompt injection, hallucination, abuse, and more.
*   [Logical Fallacy chain](https://python.langchain.com/docs/guides/productionization/safety/logical_fallacy_chain/): Checks the model output against logical fallacies to correct any deviation.
*   [Moderation chain](https://python.langchain.com/docs/guides/productionization/safety/moderation/): Check if any output text is harmful and flag it.
*   [Presidio data anonymization](https://python.langchain.com/docs/guides/productionization/safety/presidio_data_anonymization/): Helps to ensure sensitive data is properly managed and governed.

* * *

#### Help us out by providing feedback on this documentation page:
The [Layerup Security](https://uselayerup.com/) integration allows you to secure your calls to any LangChain LLM, LLM chain or LLM agent. The LLM object wraps around any existing LLM object, allowing for a secure layer between your users and your LLMs.

While the Layerup Security object is designed as an LLM, it is not actually an LLM itself, it simply wraps around an LLM, allowing it to adapt the same functionality as the underlying LLM.

Next, create a project via the [dashboard](https://dashboard.uselayerup.com/), and copy your API key. We recommend putting your API key in your project's environment.

```
from langchain_community.llms.layerup_security import LayerupSecurityfrom langchain_openai import OpenAI# Create an instance of your favorite LLMopenai = OpenAI(    model_name="gpt-3.5-turbo",    openai_api_key="OPENAI_API_KEY",)# Configure Layerup Securitylayerup_security = LayerupSecurity(    # Specify a LLM that Layerup Security will wrap around    llm=openai,    # Layerup API key, from the Layerup dashboard    layerup_api_key="LAYERUP_API_KEY",    # Custom base URL, if self hosting    layerup_api_base_url="https://api.uselayerup.com/v1",    # List of guardrails to run on prompts before the LLM is invoked    prompt_guardrails=[],    # List of guardrails to run on responses from the LLM    response_guardrails=["layerup.hallucination"],    # Whether or not to mask the prompt for PII & sensitive data before it is sent to the LLM    mask=False,    # Metadata for abuse tracking, customer tracking, and scope tracking.    metadata={"customer": "example@uselayerup.com"},    # Handler for guardrail violations on the prompt guardrails    handle_prompt_guardrail_violation=(        lambda violation: {            "role": "assistant",            "content": (                "There was sensitive data! I cannot respond. "                "Here's a dynamic canned response. Current date: {}"            ).format(datetime.now())        }        if violation["offending_guardrail"] == "layerup.sensitive_data"        else None    ),    # Handler for guardrail violations on the response guardrails    handle_response_guardrail_violation=(        lambda violation: {            "role": "assistant",            "content": (                "Custom canned response with dynamic data! "                "The violation rule was {}."            ).format(violation["offending_guardrail"])        }    ),)response = layerup_security.invoke(    "Summarize this message: my name is Bob Dylan. My SSN is 123-45-6789.")
```
## Moderation chain

This notebook walks through examples of how to use a moderation chain, and several common ways for doing so. Moderation chains are useful for detecting text that could be hateful, violent, etc. This can be useful to apply on both user input, but also on the output of a Language Model. Some API providers specifically prohibit you, or your end users, from generating some types of harmful content. To comply with this (and to just generally prevent your application from being harmful) you may want to add a moderation chain to your sequences in order to make sure any output the LLM generates is not harmful.

If the content passed into the moderation chain is harmful, there is not one best way to handle it. It probably depends on your application. Sometimes you may want to throw an error (and have your application handle that). Other times, you may want to return something to the user explaining that the text was harmful.

```
%pip install --upgrade --quiet  langchain langchain-openai
```

```
from langchain.chains import OpenAIModerationChainfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import OpenAI
```

```
moderate = OpenAIModerationChain()
```

```
model = OpenAI()prompt = ChatPromptTemplate.from_messages([("system", "repeat after me: {input}")])
```

```
chain.invoke({"input": "you are stupid"})
```

```
moderated_chain = chain | moderate
```

```
moderated_chain.invoke({"input": "you are stupid"})
```

```
{'input': '\n\nYou are stupid', 'output': "Text was found that violates OpenAI's content policy."}
```
## Passing data through

RunnablePassthrough on its own allows you to pass inputs unchanged. This typically is used in conjuction with RunnableParallel to pass data through to a new key in the map.

See the example below:

```
%pip install --upgrade --quiet  langchain langchain-openai
```

```
from langchain_core.runnables import RunnableParallel, RunnablePassthroughrunnable = RunnableParallel(    passed=RunnablePassthrough(),    modified=lambda x: x["num"] + 1,)runnable.invoke({"num": 1})
```

```
{'passed': {'num': 1}, 'extra': {'num': 1, 'mult': 3}, 'modified': 2}
```

As seen above, `passed` key was called with `RunnablePassthrough()` and so it simply passed on `{'num': 1}`.

We also set a second key in the map with `modified`. This uses a lambda to set a single value adding 1 to the num, which resulted in `modified` key with the value of `2`.

## Retrieval Example[​](#retrieval-example "Direct link to Retrieval Example")

In the example below, we see a use case where we use `RunnablePassthrough` along with `RunnableParallel`.

```
from langchain_community.vectorstores import FAISSfrom langchain_core.output_parsers import StrOutputParserfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.runnables import RunnablePassthroughfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsvectorstore = FAISS.from_texts(    ["harrison worked at kensho"], embedding=OpenAIEmbeddings())retriever = vectorstore.as_retriever()template = """Answer the question based only on the following context:{context}Question: {question}"""prompt = ChatPromptTemplate.from_template(template)model = ChatOpenAI()retrieval_chain = (    {"context": retriever, "question": RunnablePassthrough()}    | prompt    | model    | StrOutputParser())retrieval_chain.invoke("where did harrison work?")
```

```
'Harrison worked at Kensho.'
```

Here the input to prompt is expected to be a map with keys “context” and “question”. The user input is just the question. So we need to get the context using our retriever and passthrough the user input under the “question” key. In this case, the RunnablePassthrough allows us to pass on the user’s question to the prompt and model.
## Installation

## Official release[​](#official-release "Direct link to Official release")

To install LangChain run:

*   Pip
*   Conda

This will install the bare minimum requirements of LangChain. A lot of the value of LangChain comes when integrating it with various model providers, datastores, etc. By default, the dependencies needed to do that are NOT installed. You will need to install the dependencies for specific integrations separately.

## From source[​](#from-source "Direct link to From source")

If you want to install from source, you can do so by cloning the repo and be sure that the directory is `PATH/TO/REPO/langchain/libs/langchain` running:

## LangChain core[​](#langchain-core "Direct link to LangChain core")

The `langchain-core` package contains base abstractions that the rest of the LangChain ecosystem uses, along with the LangChain Expression Language. It is automatically installed by `langchain`, but can also be used separately. Install with:

```
pip install langchain-core
```

The `langchain-community` package contains third-party integrations. It is automatically installed by `langchain`, but can also be used separately. Install with:

```
pip install langchain-community
```

## LangChain experimental[​](#langchain-experimental "Direct link to LangChain experimental")

The `langchain-experimental` package holds experimental LangChain code, intended for research and experimental uses. Install with:

```
pip install langchain-experimental
```

## LangGraph[​](#langgraph "Direct link to LangGraph")

`langgraph` is a library for building stateful, multi-actor applications with LLMs, built on top of (and intended to be used with) LangChain. Install with:

## LangServe[​](#langserve "Direct link to LangServe")

LangServe helps developers deploy LangChain runnables and chains as a REST API. LangServe is automatically installed by LangChain CLI. If not using LangChain CLI, install with:

```
pip install "langserve[all]"
```

for both client and server dependencies. Or `pip install "langserve[client]"` for client code, and `pip install "langserve[server]"` for server code.

## LangChain CLI[​](#langchain-cli "Direct link to LangChain CLI")

The LangChain CLI is useful for working with LangChain templates and other LangServe projects. Install with:

```
pip install langchain-cli
```

## LangSmith SDK[​](#langsmith-sdk "Direct link to LangSmith SDK")

The LangSmith SDK is automatically installed by LangChain. If not using LangChain, install with:
## Extending LangChain

Extending LangChain's base abstractions, whether you're planning to contribute back to the open-source repo or build a bespoke internal integration, is encouraged.

Check out these guides for building your own custom classes for the following modules:

*   [Chat models](https://python.langchain.com/docs/modules/model_io/chat/custom_chat_model/) for interfacing with chat-tuned language models.
*   [LLMs](https://python.langchain.com/docs/modules/model_io/llms/custom_llm/) for interfacing with text language models.
*   [Output parsers](https://python.langchain.com/docs/modules/model_io/output_parsers/custom/) for handling language model outputs.

* * *

#### Help us out by providing feedback on this documentation page:
## Pydantic compatibility

*   Pydantic v2 was released in June, 2023 ([https://docs.pydantic.dev/2.0/blog/pydantic-v2-final/](https://docs.pydantic.dev/2.0/blog/pydantic-v2-final/))
*   v2 contains has a number of breaking changes ([https://docs.pydantic.dev/2.0/migration/](https://docs.pydantic.dev/2.0/migration/))
*   Pydantic v2 and v1 are under the same package name, so both versions cannot be installed at the same time

## LangChain Pydantic migration plan[​](#langchain-pydantic-migration-plan "Direct link to LangChain Pydantic migration plan")

As of `langchain>=0.0.267`, LangChain will allow users to install either Pydantic V1 or V2.

*   Internally LangChain will continue to [use V1](https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features).
*   During this time, users can pin their pydantic version to v1 to avoid breaking changes, or start a partial migration using pydantic v2 throughout their code, but avoiding mixing v1 and v2 code for LangChain (see below).

User can either pin to pydantic v1, and upgrade their code in one go once LangChain has migrated to v2 internally, or they can start a partial migration to v2, but must avoid mixing v1 and v2 code for LangChain.

Below are two examples of showing how to avoid mixing pydantic v1 and v2 code in the case of inheritance and in the case of passing objects to LangChain.

**Example 1: Extending via inheritance**

**YES**

```
from pydantic.v1 import root_validator, validatorclass CustomTool(BaseTool): # BaseTool is v1 code    x: int = Field(default=1)    def _run(*args, **kwargs):        return "hello"    @validator('x') # v1 code    @classmethod    def validate_x(cls, x: int) -> int:        return 1    CustomTool(    name='custom_tool',    description="hello",    x=1,)
```

Mixing Pydantic v2 primitives with Pydantic v1 primitives can raise cryptic errors

**NO**

```
from pydantic import Field, field_validator # pydantic v2class CustomTool(BaseTool): # BaseTool is v1 code    x: int = Field(default=1)    def _run(*args, **kwargs):        return "hello"    @field_validator('x') # v2 code    @classmethod    def validate_x(cls, x: int) -> int:        return 1    CustomTool(     name='custom_tool',    description="hello",    x=1,)
```

**Example 2: Passing objects to LangChain**

**YES**

```
from langchain_core.tools import Toolfrom pydantic.v1 import BaseModel, Field # <-- Uses v1 namespaceclass CalculatorInput(BaseModel):    question: str = Field()Tool.from_function( # <-- tool uses v1 namespace    func=lambda question: 'hello',    name="Calculator",    description="useful for when you need to answer questions about math",    args_schema=CalculatorInput)
```

**NO**

```
from langchain_core.tools import Toolfrom pydantic import BaseModel, Field # <-- Uses v2 namespaceclass CalculatorInput(BaseModel):    question: str = Field()Tool.from_function( # <-- tool uses v1 namespace    func=lambda question: 'hello',    name="Calculator",    description="useful for when you need to answer questions about math",    args_schema=CalculatorInput)
```
## Productionization

After you've developed a prototype of your language model application, the next step is to prepare it for production. This section contains guides around best practices for getting and keeping your application production-ready, ensuring it's ready for real-world use.

[

## 🗃️ Deployment

1 items

](https://python.langchain.com/docs/guides/productionization/deployments/)

[

## 🗃️ Evaluation

4 items

](https://python.langchain.com/docs/guides/productionization/evaluation/)

[

## 📄️ Fallbacks

When working with language models, you may often encounter issues from

](https://python.langchain.com/docs/guides/productionization/fallbacks/)

[

## 🗃️ Privacy & Safety

7 items

](https://python.langchain.com/docs/guides/productionization/safety/)

* * *

#### Help us out by providing feedback on this documentation page:
## Comparison Evaluators

Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.

These evaluators inherit from the `PairwiseStringEvaluator` class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.

To create a custom comparison evaluator, inherit from the `PairwiseStringEvaluator` class and overwrite the `_evaluate_string_pairs` method. If you require asynchronous evaluation, also overwrite the `_aevaluate_string_pairs` method.

Here's a summary of the key methods and properties of a comparison evaluator:

*   `evaluate_string_pairs`: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators.
*   `aevaluate_string_pairs`: Asynchronously evaluate the output string pairs. This function should be overwritten for asynchronous evaluation.
*   `requires_input`: This property indicates whether this evaluator requires an input string.
*   `requires_reference`: This property specifies whether this evaluator requires a reference label.

LangSmith Support

The [run\_on\_dataset](https://api.python.langchain.com/en/latest/langchain_api_reference.html#module-langchain.smith) evaluation method is designed to evaluate only a single model at a time, and thus, doesn't support these evaluators.

Detailed information about creating custom evaluators and the available built-in comparison evaluators is provided in the following sections.

[

## 📄️ Pairwise string comparison

Open In Colab

](https://python.langchain.com/docs/guides/productionization/evaluation/comparison/pairwise_string/)

[

## 📄️ Pairwise embedding distance

Open In Colab

](https://python.langchain.com/docs/guides/productionization/evaluation/comparison/pairwise_embedding_distance/)

[

## 📄️ Custom pairwise evaluator

Open In Colab

](https://python.langchain.com/docs/guides/productionization/evaluation/comparison/custom/)
ChatVectorDBChainConversationalRetrievalChainMore general to all retrieverscreate\_ernie\_fn\_chaincreate\_ernie\_fn\_runnableUse LCEL under the hoodcreated\_structured\_output\_chaincreate\_structured\_output\_runnableUse LCEL under the hoodNatBotChainNot usedcreate\_openai\_fn\_chaincreate\_openai\_fn\_runnableUse LCEL under the hoodcreate\_structured\_output\_chaincreate\_structured\_output\_runnableUse LCEL under the hoodload\_query\_constructor\_chainload\_query\_constructor\_runnableUse LCEL under the hoodVectorDBQARetrievalQAMore general to all retrieversSequential ChainLCELObviated by LCELSimpleSequentialChainLCELObviated by LCELTransformChainLCEL/RunnableLambdaObviated by LCELcreate\_tagging\_chaincreate\_structured\_output\_runnableUse LCEL under the hoodChatAgentcreate\_react\_agentUse LCEL builder over a classConversationalAgentcreate\_react\_agentUse LCEL builder over a classConversationalChatAgentcreate\_json\_chat\_agentUse LCEL builder over a classinitialize\_agentIndividual create agent methodsIndividual create agent methods are more clearZeroShotAgentcreate\_react\_agentUse LCEL builder over a classOpenAIFunctionsAgentcreate\_openai\_functions\_agentUse LCEL builder over a classOpenAIMultiFunctionsAgentcreate\_openai\_tools\_agentUse LCEL builder over a classSelfAskWithSearchAgentcreate\_self\_ask\_with\_searchUse LCEL builder over a classStructuredChatAgentcreate\_structured\_chat\_agentUse LCEL builder over a classXMLAgentcreate\_xml\_agentUse LCEL builder over a class
## Frequently Asked Questions

## Pull Requests (PRs)[​](#pull-requests-prs "Direct link to Pull Requests (PRs)")

### How do I allow maintainers to edit my PR?[​](#how-do-i-allow-maintainers-to-edit-my-pr "Direct link to How do I allow maintainers to edit my PR?")

When you submit a pull request, there may be additional changes necessary before merging it. Oftentimes, it is more efficient for the maintainers to make these changes themselves before merging, rather than asking you to do so in code review.

By default, most pull requests will have a `✅ Maintainers are allowed to edit this pull request.` badge in the right-hand sidebar.

If you do not see this badge, you may have this setting off for the fork you are pull-requesting from. See [this Github docs page](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork) for more information.

Notably, Github doesn't allow this setting to be enabled for forks in **organizations** ([issue](https://github.com/orgs/community/discussions/5634)). If you are working in an organization, we recommend submitting your PR from a personal fork in order to enable this setting.
If you plan on contributing to LangChain code or documentation, it can be useful to understand the high level structure of the repository.

```
.├── cookbook # Tutorials and examples├── docs # Contains content for the documentation here: https://python.langchain.com/├── libs│   ├── langchain # Main package│   │   ├── tests/unit_tests # Unit tests (present in each package not shown for brevity)│   │   ├── tests/integration_tests # Integration tests (present in each package not shown for brevity)│   ├── langchain-community # Third-party integrations│   ├── langchain-core # Base interfaces for key abstractions│   ├── langchain-experimental # Experimental components and chains│   ├── partners│       ├── langchain-partner-1│       ├── langchain-partner-2│       ├── ...│├── templates # A collection of easily deployable reference architectures for a wide variety of tasks.
```

There are other files in the root directory level, but their presence should be self-explanatory. Feel free to browse around!

See the [documentation](https://python.langchain.com/docs/contributing/documentation/style_guide/) guidelines to learn how to contribute to the documentation.

The `/libs` directory contains the code for the LangChain packages.
## Code writing

Example of how to use LCEL to write Python code.

```
%pip install --upgrade --quiet  langchain-core langchain-experimental langchain-openai
```

```
from langchain_core.output_parsers import StrOutputParserfrom langchain_core.prompts import (    ChatPromptTemplate,)from langchain_experimental.utilities import PythonREPLfrom langchain_openai import ChatOpenAI
```

```
template = """Write some python code to solve the user's problem. Return only python code in Markdown format, e.g.:```python....```"""prompt = ChatPromptTemplate.from_messages([("system", template), ("human", "{input}")])model = ChatOpenAI()
```

```
def _sanitize_output(text: str):    _, after = text.split("```python")    return after.split("```")[0]
```

```
chain = prompt | model | StrOutputParser() | _sanitize_output | PythonREPL().run
```

```
chain.invoke({"input": "whats 2 plus 2"})
```

```
Python REPL can execute arbitrary code. Use with caution.
```

* * *

#### Help us out by providing feedback on this documentation page:
You can also turn an arbitrary function into a chain by adding a `@chain` decorator. This is functionaly equivalent to wrapping in a [`RunnableLambda`](https://python.langchain.com/docs/expression_language/primitives/functions/).

This will have the benefit of improved observability by tracing your chain correctly. Any calls to runnables inside this function will be traced as nested childen.

It will also allow you to use this as any other runnable, compose it in chain, etc.

Let’s take a look at this in action!

```
%pip install --upgrade --quiet  langchain langchain-openai
```

```
from langchain_core.output_parsers import StrOutputParserfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_core.runnables import chainfrom langchain_openai import ChatOpenAI
```

```
prompt1 = ChatPromptTemplate.from_template("Tell me a joke about {topic}")prompt2 = ChatPromptTemplate.from_template("What is the subject of this joke: {joke}")
```

```
@chaindef custom_chain(text):    prompt_val1 = prompt1.invoke({"topic": text})    output1 = ChatOpenAI().invoke(prompt_val1)    parsed_output1 = StrOutputParser().invoke(output1)    chain2 = prompt2 | ChatOpenAI() | StrOutputParser()    return chain2.invoke({"joke": parsed_output1})
```

`custom_chain` is now a runnable, meaning you will need to use `invoke`

```
custom_chain.invoke("bears")
```

```
'The subject of this joke is bears.'
```

If you check out your LangSmith traces, you should see a `custom_chain` trace in there, with the calls to OpenAI nested underneath
## Primitives

In addition to various [components](https://python.langchain.com/docs/modules/) that are usable with LCEL, LangChain also includes various primitives that help pass around and format data, bind arguments, invoke custom logic, and more.

This section goes into greater depth on where and how some of these components are useful.

[

## 📄️ Sequences: Chaining runnables

chaining-runnables}

](https://python.langchain.com/docs/expression_language/primitives/sequence/)

[

## 📄️ Parallel: Format data

formatting-inputs-output}

](https://python.langchain.com/docs/expression_language/primitives/parallel/)

[

## 📄️ Binding: Attach runtime args

binding-attach-runtime-args}

](https://python.langchain.com/docs/expression_language/primitives/binding/)

[

## 📄️ Lambda: Run custom functions

run-custom-functions}

](https://python.langchain.com/docs/expression_language/primitives/functions/)

[

## 📄️ Passthrough: Pass through inputs

passing-data-through}

](https://python.langchain.com/docs/expression_language/primitives/passthrough/)

[

## 📄️ Assign: Add values to state

adding-values-to-chain-state}

](https://python.langchain.com/docs/expression_language/primitives/assign/)

[

## 📄️ Configure runtime chain internals

configure-chain-internals-at-runtime}

](https://python.langchain.com/docs/expression_language/primitives/configure/)
**Please ensure OpenAI library is version 1.0.0 or higher; otherwise, refer to the older doc [OpenAI Adapter(Old)](https://python.langchain.com/docs/integrations/adapters/openai-old/).**

A lot of people get started with OpenAI but want to explore other models. LangChain’s integrations with many model providers make this easy to do so. While LangChain has it’s own message and model APIs, we’ve also made it as easy as possible to explore other models by exposing an adapter to adapt LangChain models to the OpenAI api.

At the moment this only deals with output and does not return other information (token counts, stop reasons, etc).

```
{'content': 'Hello! How can I assist you today?', 'role': 'assistant', 'function_call': None, 'tool_calls': None}
```

```
{'role': 'assistant', 'content': 'Hello! How can I help you today?'}
```

```
{'role': 'assistant', 'content': 'Hello! How can I help you today?'}
```

```
{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}
```

```
{'content': '', 'function_call': None, 'role': 'assistant', 'tool_calls': None}{'content': 'Hello', 'function_call': None, 'role': None, 'tool_calls': None}{'content': '!', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' How', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' can', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' I', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' assist', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' you', 'function_call': None, 'role': None, 'tool_calls': None}{'content': ' today', 'function_call': None, 'role': None, 'tool_calls': None}{'content': '?', 'function_call': None, 'role': None, 'tool_calls': None}{'content': None, 'function_call': None, 'role': None, 'tool_calls': None}
```

```
{'role': 'assistant', 'content': ''}{'content': 'Hello'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}{'content': '?'}{}
```

```
{'role': 'assistant', 'content': ''}{'content': 'Hello'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}{'content': '?'}{}
```
## OpenAI Adapter(Old)

**Please ensure OpenAI library is less than 1.0.0; otherwise, refer to the newer doc [OpenAI Adapter](https://python.langchain.com/docs/integrations/adapters/openai/).**

A lot of people get started with OpenAI but want to explore other models. LangChain’s integrations with many model providers make this easy to do so. While LangChain has it’s own message and model APIs, we’ve also made it as easy as possible to explore other models by exposing an adapter to adapt LangChain models to the OpenAI api.

At the moment this only deals with output and does not return other information (token counts, stop reasons, etc).

```
import openaifrom langchain_community.adapters import openai as lc_openai
```

## ChatCompletion.create[​](#chatcompletion.create "Direct link to ChatCompletion.create")

```
messages = [{"role": "user", "content": "hi"}]
```

Original OpenAI call

```
result = openai.ChatCompletion.create(    messages=messages, model="gpt-3.5-turbo", temperature=0)result["choices"][0]["message"].to_dict_recursive()
```

```
{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}
```

LangChain OpenAI wrapper call

```
lc_result = lc_openai.ChatCompletion.create(    messages=messages, model="gpt-3.5-turbo", temperature=0)lc_result["choices"][0]["message"]
```

```
{'role': 'assistant', 'content': 'Hello! How can I assist you today?'}
```

Swapping out model providers

```
lc_result = lc_openai.ChatCompletion.create(    messages=messages, model="claude-2", temperature=0, provider="ChatAnthropic")lc_result["choices"][0]["message"]
```

```
{'role': 'assistant', 'content': ' Hello!'}
```

## ChatCompletion.stream[​](#chatcompletion.stream "Direct link to ChatCompletion.stream")

Original OpenAI call

```
for c in openai.ChatCompletion.create(    messages=messages, model="gpt-3.5-turbo", temperature=0, stream=True):    print(c["choices"][0]["delta"].to_dict_recursive())
```

```
{'role': 'assistant', 'content': ''}{'content': 'Hello'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}{'content': '?'}{}
```

LangChain OpenAI wrapper call

```
for c in lc_openai.ChatCompletion.create(    messages=messages, model="gpt-3.5-turbo", temperature=0, stream=True):    print(c["choices"][0]["delta"])
```

```
{'role': 'assistant', 'content': ''}{'content': 'Hello'}{'content': '!'}{'content': ' How'}{'content': ' can'}{'content': ' I'}{'content': ' assist'}{'content': ' you'}{'content': ' today'}{'content': '?'}{}
```

Swapping out model providers

```
for c in lc_openai.ChatCompletion.create(    messages=messages,    model="claude-2",    temperature=0,    stream=True,    provider="ChatAnthropic",):    print(c["choices"][0]["delta"])
```

```
{'role': 'assistant', 'content': ' Hello'}{'content': '!'}{}
```
## Comet Tracing

There are two ways to trace your LangChains executions with Comet:

1.  Setting the `LANGCHAIN_COMET_TRACING` environment variable to “true”. This is the recommended way.
2.  Import the `CometTracer` manually and pass it explicitely.

```
import osimport comet_llmos.environ["LANGCHAIN_COMET_TRACING"] = "true"# Connect to Comet if no API Key is setcomet_llm.init()# comet documentation to configure comet using env variables# https://www.comet.com/docs/v2/api-and-sdk/llm-sdk/configuration/# here we are configuring the comet projectos.environ["COMET_PROJECT_NAME"] = "comet-example-langchain-tracing"from langchain.agents import AgentType, initialize_agent, load_toolsfrom langchain.llms import OpenAI
```

```
# Agent run with tracing. Ensure that OPENAI_API_KEY is set appropriately to run this example.llm = OpenAI(temperature=0)tools = load_tools(["llm-math"], llm=llm)
```

```
agent = initialize_agent(    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)agent.run("What is 2 raised to .123243 power?")  # this should be traced# An url for the chain like the following should print in your console:# https://www.comet.com/<workspace>/<project_name># The url can be used to view the LLM chain in Comet.
```

```
# Now, we unset the environment variable and use a context manager.if "LANGCHAIN_COMET_TRACING" in os.environ:    del os.environ["LANGCHAIN_COMET_TRACING"]from langchain_community.callbacks.tracers.comet import CometTracertracer = CometTracer()# Recreate the LLM, tools and agent and passing the callback to each of themllm = OpenAI(temperature=0)tools = load_tools(["llm-math"], llm=llm)agent = initialize_agent(    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)agent.run(    "What is 2 raised to .123243 power?", callbacks=[tracer])  # this should be traced
```

* * *

#### Help us out by providing feedback on this documentation page:
[Fiddler](https://www.fiddler.ai/) is the pioneer in enterprise Generative and Predictive system ops, offering a unified platform that enables Data Science, MLOps, Risk, Compliance, Analytics, and other LOB teams to monitor, explain, analyze, and improve ML deployments at enterprise scale.

These can be found by navigating to the _Settings_ page of your Fiddler environment.

```
URL = ""  # Your Fiddler instance URL, Make sure to include the full URL (including https://). For example: https://demo.fiddler.aiORG_NAME = ""AUTH_TOKEN = ""  # Your Fiddler instance auth token# Fiddler project and model names, used for model registrationPROJECT_NAME = ""MODEL_NAME = ""  # Model name in Fiddler
```

```
from langchain_community.callbacks.fiddler_callback import FiddlerCallbackHandlerfiddler_handler = FiddlerCallbackHandler(    url=URL,    org=ORG_NAME,    project=PROJECT_NAME,    model=MODEL_NAME,    api_key=AUTH_TOKEN,)
```

```
from langchain_core.output_parsers import StrOutputParserfrom langchain_openai import OpenAI# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEYllm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])output_parser = StrOutputParser()chain = llm | output_parser# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generatedchain.invoke("How far is moon from earth?")
```

```
# Few more invocationschain.invoke("What is the temperature on Mars?")chain.invoke("How much is 2 + 200000?")chain.invoke("Which movie won the oscars this year?")chain.invoke("Can you write me a poem about insomnia?")chain.invoke("How are you doing today?")chain.invoke("What is the meaning of life?")
```

```
from langchain_core.prompts import (    ChatPromptTemplate,    FewShotChatMessagePromptTemplate,)examples = [    {"input": "2+2", "output": "4"},    {"input": "2+3", "output": "5"},]example_prompt = ChatPromptTemplate.from_messages(    [        ("human", "{input}"),        ("ai", "{output}"),    ])few_shot_prompt = FewShotChatMessagePromptTemplate(    example_prompt=example_prompt,    examples=examples,)final_prompt = ChatPromptTemplate.from_messages(    [        ("system", "You are a wondrous wizard of math."),        few_shot_prompt,        ("human", "{input}"),    ])# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEYllm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])chain = final_prompt | llm# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generatedchain.invoke({"input": "What's the square of a triangle?"})
```