!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Get started](/v0.1/docs/get_started/)
    
    *   [Introduction](/v0.1/docs/get_started/introduction/)
    *   [Installation](/v0.1/docs/get_started/installation/)
    *   [Quickstart](/v0.1/docs/get_started/quickstart/)
*   [LangChain Expression Language](/v0.1/docs/expression_language/)
    
    *   [Get started](/v0.1/docs/expression_language/get_started/)
    *   [Why use LCEL?](/v0.1/docs/expression_language/why/)
    *   [Interface](/v0.1/docs/expression_language/interface/)
    *   [Streaming](/v0.1/docs/expression_language/streaming/)
    *   [How to](/v0.1/docs/expression_language/how_to/routing/)
        
    *   [Cookbook](/v0.1/docs/expression_language/cookbook/)
        
    *   [LangChain Expression Language (LCEL)](/v0.1/docs/expression_language/)
*   [Modules](/v0.1/docs/modules/)
    
    *   [Model I/O](/v0.1/docs/modules/model_io/)
        
    *   [Retrieval](/v0.1/docs/modules/data_connection/)
        
    *   [Chains](/v0.1/docs/modules/chains/)
    *   [Agents](/v0.1/docs/modules/agents/)
        
    *   [More](/v0.1/docs/modules/memory/)
        
*   [Security](/v0.1/docs/security/)
*   [Guides](/v0.1/docs/guides/)
    
    *   [Debugging](/v0.1/docs/guides/debugging/)
    *   [Deployment](/v0.1/docs/guides/deployment/)
        
    *   [Evaluation](/v0.1/docs/guides/evaluation/)
        
        *   [String Evaluators](/v0.1/docs/guides/evaluation/string/)
            
        *   [Comparison Evaluators](/v0.1/docs/guides/evaluation/comparison/)
            
            *   [Pairwise Embedding Distance](/v0.1/docs/guides/evaluation/comparison/pairwise_embedding_distance/)
            *   [Pairwise String Comparison](/v0.1/docs/guides/evaluation/comparison/pairwise_string/)
        *   [Trajectory Evaluators](/v0.1/docs/guides/evaluation/trajectory/)
            
        *   [Examples](/v0.1/docs/guides/evaluation/examples/)
            
    *   [Extending LangChain.js](/v0.1/docs/guides/extending_langchain/)
    *   [Fallbacks](/v0.1/docs/guides/fallbacks/)
    *   [LangSmith Walkthrough](/v0.1/docs/guides/langsmith_evaluation/)
    *   [Migrating to 0.1](/v0.1/docs/guides/migrating/)
*   [Ecosystem](/v0.1/docs/ecosystem/)
    
*   [LangGraph](/v0.1/docs/langgraph/)
*   * * *
    

*   [](/v0.1/)
*   [Guides](/v0.1/docs/guides/)
*   [Evaluation](/v0.1/docs/guides/evaluation/)
*   Comparison Evaluators

Comparison Evaluators
=====================

Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.

These evaluators inherit from the `PairwiseStringEvaluator` or `LLMPairwiseStringEvaluator` class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.

To create a custom comparison evaluator, inherit from the `PairwiseStringEvaluator` or `LLMPairwiseStringEvaluator` abstract classes exported from `langchain/evaluation` and overwrite the `_evaluateStringPairs` method.

Here's a summary of the key methods and properties of a comparison evaluator:

*   `_evaluateStringPairs`: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators.
*   `requiresInput`: This property indicates whether this evaluator requires an input string.
*   `requiresReference`: This property specifies whether this evaluator requires a reference label.

Detailed information about creating custom evaluators and the available built-in comparison evaluators is provided in the following sections.

[

📄️ Pairwise Embedding Distance
-------------------------------

One way to measure the similarity (or dissimilarity) between two predictions on a shared or similar input is to embed the predictions and compute a vector distance between the two embeddings.

](/v0.1/docs/guides/evaluation/comparison/pairwise_embedding_distance/)

[

📄️ Pairwise String Comparison
------------------------------

Often you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:

](/v0.1/docs/guides/evaluation/comparison/pairwise_string/)

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

Embedding Distance

](/v0.1/docs/guides/evaluation/string/embedding_distance/)[

Next

Pairwise Embedding Distance

](/v0.1/docs/guides/evaluation/comparison/pairwise_embedding_distance/)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.