The output parser used to parse the output.
param return_reasoning: bool = False¶
Whether to return the reasoning along with the score.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only


addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Asynchronously execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.


Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async aevaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶
Asynchronously evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
Returns
The evaluation result.
Return type
dict
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Call the chain on all inputs in the list.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.


sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
**kwargs – Keyword arguments passed to default pydantic.BaseModel.dict
method.
Returns
A dictionary representation of the chain.
Example
..code-block:: python
chain.dict(exclude_unset=True)
# -> {“_type”: “foo”, “verbose”: False, …}
evaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶
Evaluate a trajectory.
Parameters


Evaluate a trajectory.
Parameters
prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.
Returns
The evaluation result.
Return type
dict
classmethod from_llm(llm: BaseLanguageModel, agent_tools: Optional[Sequence[BaseTool]] = None, output_parser: Optional[TrajectoryOutputParser] = None, return_reasoning: bool = False, **kwargs: Any) → TrajectoryEvalChain[source]¶
Create a TrajectoryEvalChain object from a language model chain.
Parameters
llm (BaseChatModel) – The language model chain.
agent_tools (Optional[Sequence[BaseTool]]) – A list of tools
available tothe agent.
output_parser (Optional[TrajectoryOutputParser]) – The output parser
used to parse the chain output into a score.
return_reasoning (bool) – Whether to return the
reasoning along with the score.
Returns
The TrajectoryEvalChain object.
Return type
TrajectoryEvalChain
static get_agent_trajectory(steps: Union[str, Sequence[Tuple[AgentAction, str]]]) → str[source]¶
Get the agent trajectory as a formatted string.
Parameters
steps (Union[str, List[Tuple[AgentAction, str]]]) – The agent trajectory.
Returns
The formatted agent trajectory.
Return type
str
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str][source]¶
Validate and prep inputs.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prepare chain outputs, and save info about this run to memory.
Parameters


Validate and prepare chain outputs, and save info about this run to memory.
Parameters
inputs – Dictionary of chain inputs, including any inputs added by chain
memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False,
inputs are also added to the final outputs.
Returns
A dict of the final chain outputs.
validator raise_callback_manager_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."
save(file_path: Union[Path, str]) → None¶
Save the chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
file_path – Path to file to save the chain to.
Example
chain.save(file_path="path/chain.yaml")
validator set_verbose  »  verbose¶
Set the chain verbosity.
Defaults to the global setting if not specified by the user.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property input_keys: List[str]¶
Get the input keys for the chain.
Returns
The input keys.
Return type
List[str]
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶


property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
property output_keys: List[str]¶
Get the output keys for the chain.
Returns
The output keys.
Return type
List[str]
property requires_input: bool¶
Whether this evaluator requires an input string.
property requires_reference: bool¶
Whether this evaluator requires a reference label.
model Config[source]¶
Bases: object
Configuration for the QAEvalChain.
extra = 'ignore'¶


langchain.evaluation.embedding_distance.base.EmbeddingDistance¶
class langchain.evaluation.embedding_distance.base.EmbeddingDistance(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶
Bases: str, Enum
Embedding Distance Metric.
COSINE¶
Cosine distance metric.
EUCLIDEAN¶
Euclidean distance metric.
MANHATTAN¶
Manhattan distance metric.
CHEBYSHEV¶
Chebyshev distance metric.
HAMMING¶
Hamming distance metric.
Methods
__init__(*args, **kwds)
capitalize()
Return a capitalized version of the string.
casefold()
Return a version of the string suitable for caseless comparisons.
center(width[, fillchar])
Return a centered string of length width.
count(sub[, start[, end]])
Return the number of non-overlapping occurrences of substring sub in string S[start:end].
encode([encoding, errors])
Encode the string using the codec registered for encoding.
endswith(suffix[, start[, end]])
Return True if S ends with the specified suffix, False otherwise.
expandtabs([tabsize])
Return a copy where all tab characters are expanded using spaces.
find(sub[, start[, end]])
Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end].
format(*args, **kwargs)
Return a formatted version of S, using substitutions from args and kwargs.
format_map(mapping)
Return a formatted version of S, using substitutions from mapping.
index(sub[, start[, end]])
Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end].
isalnum()


isalnum()
Return True if the string is an alpha-numeric string, False otherwise.
isalpha()
Return True if the string is an alphabetic string, False otherwise.
isascii()
Return True if all characters in the string are ASCII, False otherwise.
isdecimal()
Return True if the string is a decimal string, False otherwise.
isdigit()
Return True if the string is a digit string, False otherwise.
isidentifier()
Return True if the string is a valid Python identifier, False otherwise.
islower()
Return True if the string is a lowercase string, False otherwise.
isnumeric()
Return True if the string is a numeric string, False otherwise.
isprintable()
Return True if the string is printable, False otherwise.
isspace()
Return True if the string is a whitespace string, False otherwise.
istitle()
Return True if the string is a title-cased string, False otherwise.
isupper()
Return True if the string is an uppercase string, False otherwise.
join(iterable, /)
Concatenate any number of strings.
ljust(width[, fillchar])
Return a left-justified string of length width.
lower()
Return a copy of the string converted to lowercase.
lstrip([chars])
Return a copy of the string with leading whitespace removed.
maketrans
Return a translation table usable for str.translate().
partition(sep, /)
Partition the string into three parts using the given separator.
removeprefix(prefix, /)
Return a str with the given prefix string removed if present.
removesuffix(suffix, /)
Return a str with the given suffix string removed if present.
replace(old, new[, count])
Return a copy with all occurrences of substring old replaced by new.
rfind(sub[, start[, end]])


rfind(sub[, start[, end]])
Return the highest index in S where substring sub is found, such that sub is contained within S[start:end].
rindex(sub[, start[, end]])
Return the highest index in S where substring sub is found, such that sub is contained within S[start:end].
rjust(width[, fillchar])
Return a right-justified string of length width.
rpartition(sep, /)
Partition the string into three parts using the given separator.
rsplit([sep, maxsplit])
Return a list of the substrings in the string, using sep as the separator string.
rstrip([chars])
Return a copy of the string with trailing whitespace removed.
split([sep, maxsplit])
Return a list of the substrings in the string, using sep as the separator string.
splitlines([keepends])
Return a list of the lines in the string, breaking at line boundaries.
startswith(prefix[, start[, end]])
Return True if S starts with the specified prefix, False otherwise.
strip([chars])
Return a copy of the string with leading and trailing whitespace removed.
swapcase()
Convert uppercase characters to lowercase and lowercase characters to uppercase.
title()
Return a version of the string where each word is titlecased.
translate(table, /)
Replace each character in the string using the given translation table.
upper()
Return a copy of the string converted to uppercase.
zfill(width, /)
Pad a numeric string with zeros on the left, to fill a field of the given width.
Attributes
COSINE
EUCLIDEAN
MANHATTAN
CHEBYSHEV
HAMMING
capitalize()¶
Return a capitalized version of the string.
More specifically, make the first character have upper case and the rest lower
case.


More specifically, make the first character have upper case and the rest lower
case.
casefold()¶
Return a version of the string suitable for caseless comparisons.
center(width, fillchar=' ', /)¶
Return a centered string of length width.
Padding is done using the specified fill character (default is a space).
count(sub[, start[, end]]) → int¶
Return the number of non-overlapping occurrences of substring sub in
string S[start:end].  Optional arguments start and end are
interpreted as in slice notation.
encode(encoding='utf-8', errors='strict')¶
Encode the string using the codec registered for encoding.
encodingThe encoding in which to encode the string.
errorsThe error handling scheme to use for encoding errors.
The default is ‘strict’ meaning that encoding errors raise a
UnicodeEncodeError.  Other possible values are ‘ignore’, ‘replace’ and
‘xmlcharrefreplace’ as well as any other name registered with
codecs.register_error that can handle UnicodeEncodeErrors.
endswith(suffix[, start[, end]]) → bool¶
Return True if S ends with the specified suffix, False otherwise.
With optional start, test S beginning at that position.
With optional end, stop comparing S at that position.
suffix can also be a tuple of strings to try.
expandtabs(tabsize=8)¶
Return a copy where all tab characters are expanded using spaces.
If tabsize is not given, a tab size of 8 characters is assumed.
find(sub[, start[, end]]) → int¶
Return the lowest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Return -1 on failure.
format(*args, **kwargs) → str¶


Return -1 on failure.
format(*args, **kwargs) → str¶
Return a formatted version of S, using substitutions from args and kwargs.
The substitutions are identified by braces (‘{’ and ‘}’).
format_map(mapping) → str¶
Return a formatted version of S, using substitutions from mapping.
The substitutions are identified by braces (‘{’ and ‘}’).
index(sub[, start[, end]]) → int¶
Return the lowest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Raises ValueError when the substring is not found.
isalnum()¶
Return True if the string is an alpha-numeric string, False otherwise.
A string is alpha-numeric if all characters in the string are alpha-numeric and
there is at least one character in the string.
isalpha()¶
Return True if the string is an alphabetic string, False otherwise.
A string is alphabetic if all characters in the string are alphabetic and there
is at least one character in the string.
isascii()¶
Return True if all characters in the string are ASCII, False otherwise.
ASCII characters have code points in the range U+0000-U+007F.
Empty string is ASCII too.
isdecimal()¶
Return True if the string is a decimal string, False otherwise.
A string is a decimal string if all characters in the string are decimal and
there is at least one character in the string.
isdigit()¶
Return True if the string is a digit string, False otherwise.
A string is a digit string if all characters in the string are digits and there
is at least one character in the string.
isidentifier()¶
Return True if the string is a valid Python identifier, False otherwise.


isidentifier()¶
Return True if the string is a valid Python identifier, False otherwise.
Call keyword.iskeyword(s) to test whether string s is a reserved identifier,
such as “def” or “class”.
islower()¶
Return True if the string is a lowercase string, False otherwise.
A string is lowercase if all cased characters in the string are lowercase and
there is at least one cased character in the string.
isnumeric()¶
Return True if the string is a numeric string, False otherwise.
A string is numeric if all characters in the string are numeric and there is at
least one character in the string.
isprintable()¶
Return True if the string is printable, False otherwise.
A string is printable if all of its characters are considered printable in
repr() or if it is empty.
isspace()¶
Return True if the string is a whitespace string, False otherwise.
A string is whitespace if all characters in the string are whitespace and there
is at least one character in the string.
istitle()¶
Return True if the string is a title-cased string, False otherwise.
In a title-cased string, upper- and title-case characters may only
follow uncased characters and lowercase characters only cased ones.
isupper()¶
Return True if the string is an uppercase string, False otherwise.
A string is uppercase if all cased characters in the string are uppercase and
there is at least one cased character in the string.
join(iterable, /)¶
Concatenate any number of strings.
The string whose method is called is inserted in between each given string.
The result is returned as a new string.
Example: ‘.’.join([‘ab’, ‘pq’, ‘rs’]) -> ‘ab.pq.rs’
ljust(width, fillchar=' ', /)¶


ljust(width, fillchar=' ', /)¶
Return a left-justified string of length width.
Padding is done using the specified fill character (default is a space).
lower()¶
Return a copy of the string converted to lowercase.
lstrip(chars=None, /)¶
Return a copy of the string with leading whitespace removed.
If chars is given and not None, remove characters in chars instead.
static maketrans()¶
Return a translation table usable for str.translate().
If there is only one argument, it must be a dictionary mapping Unicode
ordinals (integers) or characters to Unicode ordinals, strings or None.
Character keys will be then converted to ordinals.
If there are two arguments, they must be strings of equal length, and
in the resulting dictionary, each character in x will be mapped to the
character at the same position in y. If there is a third argument, it
must be a string, whose characters will be mapped to None in the result.
partition(sep, /)¶
Partition the string into three parts using the given separator.
This will search for the separator in the string.  If the separator is found,
returns a 3-tuple containing the part before the separator, the separator
itself, and the part after it.
If the separator is not found, returns a 3-tuple containing the original string
and two empty strings.
removeprefix(prefix, /)¶
Return a str with the given prefix string removed if present.
If the string starts with the prefix string, return string[len(prefix):].
Otherwise, return a copy of the original string.
removesuffix(suffix, /)¶
Return a str with the given suffix string removed if present.
If the string ends with the suffix string and that suffix is not empty,


If the string ends with the suffix string and that suffix is not empty,
return string[:-len(suffix)]. Otherwise, return a copy of the original
string.
replace(old, new, count=- 1, /)¶
Return a copy with all occurrences of substring old replaced by new.
countMaximum number of occurrences to replace.
-1 (the default value) means replace all occurrences.
If the optional argument count is given, only the first count occurrences are
replaced.
rfind(sub[, start[, end]]) → int¶
Return the highest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Return -1 on failure.
rindex(sub[, start[, end]]) → int¶
Return the highest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Raises ValueError when the substring is not found.
rjust(width, fillchar=' ', /)¶
Return a right-justified string of length width.
Padding is done using the specified fill character (default is a space).
rpartition(sep, /)¶
Partition the string into three parts using the given separator.
This will search for the separator in the string, starting at the end. If
the separator is found, returns a 3-tuple containing the part before the
separator, the separator itself, and the part after it.
If the separator is not found, returns a 3-tuple containing two empty strings
and the original string.
rsplit(sep=None, maxsplit=- 1)¶
Return a list of the substrings in the string, using sep as the separator string.
sepThe separator used to split the string.


sepThe separator used to split the string.
When set to None (the default value), will split on any whitespace
character (including \n \r \t \f and spaces) and will discard
empty strings from the result.
maxsplitMaximum number of splits (starting from the left).
-1 (the default value) means no limit.
Splitting starts at the end of the string and works to the front.
rstrip(chars=None, /)¶
Return a copy of the string with trailing whitespace removed.
If chars is given and not None, remove characters in chars instead.
split(sep=None, maxsplit=- 1)¶
Return a list of the substrings in the string, using sep as the separator string.
sepThe separator used to split the string.
When set to None (the default value), will split on any whitespace
character (including \n \r \t \f and spaces) and will discard
empty strings from the result.
maxsplitMaximum number of splits (starting from the left).
-1 (the default value) means no limit.
Note, str.split() is mainly useful for data that has been intentionally
delimited.  With natural text that includes punctuation, consider using
the regular expression module.
splitlines(keepends=False)¶
Return a list of the lines in the string, breaking at line boundaries.
Line breaks are not included in the resulting list unless keepends is given and
true.
startswith(prefix[, start[, end]]) → bool¶
Return True if S starts with the specified prefix, False otherwise.
With optional start, test S beginning at that position.
With optional end, stop comparing S at that position.
prefix can also be a tuple of strings to try.
strip(chars=None, /)¶
Return a copy of the string with leading and trailing whitespace removed.


Return a copy of the string with leading and trailing whitespace removed.
If chars is given and not None, remove characters in chars instead.
swapcase()¶
Convert uppercase characters to lowercase and lowercase characters to uppercase.
title()¶
Return a version of the string where each word is titlecased.
More specifically, words start with uppercased characters and all remaining
cased characters have lower case.
translate(table, /)¶
Replace each character in the string using the given translation table.
tableTranslation table, which must be a mapping of Unicode ordinals to
Unicode ordinals, strings, or None.
The table must implement lookup/indexing via __getitem__, for instance a
dictionary or list.  If this operation raises LookupError, the character is
left untouched.  Characters mapped to None are deleted.
upper()¶
Return a copy of the string converted to uppercase.
zfill(width, /)¶
Pad a numeric string with zeros on the left, to fill a field of the given width.
The string is never truncated.
CHEBYSHEV = 'chebyshev'¶
COSINE = 'cosine'¶
EUCLIDEAN = 'euclidean'¶
HAMMING = 'hamming'¶
MANHATTAN = 'manhattan'¶


langchain.evaluation.run_evaluators.implementations.TrajectoryInputMapper¶
class langchain.evaluation.run_evaluators.implementations.TrajectoryInputMapper(*, agent_input_key: str = 'input', agent_output_key: str = 'output', tool_input_key: str = 'input', tool_output_key: str = 'output', reference_output_key: Optional[str] = None)[source]¶
Bases: RunEvaluatorInputMapper, BaseModel
Maps the Run and Optional[Example] to a dictionary.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param agent_input_key: str = 'input'¶
The key to load from the agent executor’s run input dictionary.
param agent_output_key: str = 'output'¶
The key to load from the agent executor’s run output dictionary.
param reference_output_key: Optional[str] = None¶
The key to use for selecting the reference answer.
param tool_input_key: str = 'input'¶
The key to load from the tool executor’s run input dictionary.
param tool_output_key: str = 'output'¶
The key to load from the tool executor’s run output dictionary.
__call__(run: Run, example: Optional[Example] = None) → Any¶
Maps the Run and Optional[Example] to a dictionary
map(run: Run, example: Optional[Example] = None) → Dict[str, str][source]¶
Maps the Run and Optional[Example] to a dictionary


langchain.evaluation.qa.generate_chain.QAGenerateChain¶
class langchain.evaluation.qa.generate_chain.QAGenerateChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseLLMOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: LLMChain
LLM Chain specifically for generating examples for question answering.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.


them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param output_key: str = 'text'¶
param output_parser: BaseLLMOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶


Execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async aapply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
async aapply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.


Call apply and then parse the results.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Asynchronously execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async agenerate(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.


Generate LLM result from inputs.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Utilize the LLM generate method for speed gains.
apply_and_parse(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → Sequence[Union[str, List[str], Dict[str, str]]]¶
Call apply and then parse the results.
async apredict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
async apredict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, str]]¶
Call apredict and then parse the results.
async aprep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[AsyncCallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.


Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."


# -> "The temperature in Boise is..."
create_outputs(llm_result: LLMResult) → List[Dict[str, Any]]¶
Create outputs from response.
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
**kwargs – Keyword arguments passed to default pydantic.BaseModel.dict
method.
Returns
A dictionary representation of the chain.
Example
..code-block:: python
chain.dict(exclude_unset=True)
# -> {“_type”: “foo”, “verbose”: False, …}
classmethod from_llm(llm: BaseLanguageModel, **kwargs: Any) → QAGenerateChain[source]¶
Load QA Generate Chain from LLM.
classmethod from_string(llm: BaseLanguageModel, template: str) → LLMChain¶
Create LLMChain from LLM and template.
generate(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → LLMResult¶
Generate LLM result from inputs.
predict(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → str¶
Format prompt with kwargs and pass to LLM.
Parameters
callbacks – Callbacks to pass to LLMChain
**kwargs – Keys to pass to prompt template.
Returns
Completion from LLM.
Example
completion = llm.predict(adjective="funny")
predict_and_parse(callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, **kwargs: Any) → Union[str, List[str], Dict[str, Any]]¶
Call predict and then parse the results.
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶


Validate and prepare chain inputs, including adding inputs from memory.
Parameters
inputs – Dictionary of raw inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
Returns
A dictionary of all inputs, including those added by the chain’s memory.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶
Validate and prepare chain outputs, and save info about this run to memory.
Parameters
inputs – Dictionary of chain inputs, including any inputs added by chain
memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False,
inputs are also added to the final outputs.
Returns
A dict of the final chain outputs.
prep_prompts(input_list: List[Dict[str, Any]], run_manager: Optional[CallbackManagerForChainRun] = None) → Tuple[List[PromptValue], Optional[List[str]]]¶
Prepare prompts from inputs.
validator raise_callback_manager_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.


info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."
save(file_path: Union[Path, str]) → None¶
Save the chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
file_path – Path to file to save the chain to.
Example
chain.save(file_path="path/chain.yaml")
validator set_verbose  »  verbose¶
Set the chain verbosity.


validator set_verbose  »  verbose¶
Set the chain verbosity.
Defaults to the global setting if not specified by the user.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶
extra = 'forbid'¶


langchain.evaluation.string_distance.base.StringDistanceEvalChain¶
class langchain.evaluation.string_distance.base.StringDistanceEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, distance: StringDistance = StringDistance.LEVENSHTEIN)[source]¶
Bases: _RapidFuzzChainMixin, StringEvaluator
Compute string distances between the prediction and the reference.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param distance: langchain.evaluation.string_distance.base.StringDistance = StringDistance.LEVENSHTEIN¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None


Optional metadata associated with the chain. Defaults to None
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.


these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Asynchronously execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None


metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Call the chain on all inputs in the list.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
**kwargs – Keyword arguments passed to default pydantic.BaseModel.dict
method.
Returns
A dictionary representation of the chain.
Example
..code-block:: python
chain.dict(exclude_unset=True)
# -> {“_type”: “foo”, “verbose”: False, …}
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prepare chain inputs, including adding inputs from memory.
Parameters
inputs – Dictionary of raw inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
Returns
A dictionary of all inputs, including those added by the chain’s memory.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶


Validate and prepare chain outputs, and save info about this run to memory.
Parameters
inputs – Dictionary of chain inputs, including any inputs added by chain
memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False,
inputs are also added to the final outputs.
Returns
A dict of the final chain outputs.
validator raise_callback_manager_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."
save(file_path: Union[Path, str]) → None¶
Save the chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
file_path – Path to file to save the chain to.
Example
chain.save(file_path="path/chain.yaml")
validator set_verbose  »  verbose¶
Set the chain verbosity.
Defaults to the global setting if not specified by the user.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
validator validate_dependencies  »  all fields¶
Validate that the rapidfuzz library is installed.
Parameters
values (Dict[str, Any]) – The input values.
Returns
The validated values.
Return type
Dict[str, Any]
property evaluation_name: str¶
property input_keys: List[str]¶
Get the input keys.
Returns
The input keys.
Return type
List[str]
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the


property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
property metric: Callable¶
Get the distance metric function.
Returns
The distance metric function.
Return type
Callable
property output_keys: List[str]¶
Get the output keys.
Returns
The output keys.
Return type
List[str]
property requires_input: bool¶
Check if input is required.
Returns
True if input is required, False otherwise.
Return type
bool
property requires_reference: bool¶
Check if reference is required.
Returns
True if reference is required, False otherwise.
Return type
bool
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶


langchain.evaluation.schema.EvaluatorType¶
class langchain.evaluation.schema.EvaluatorType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶
Bases: str, Enum
The types of the evaluators.
Methods
__init__(*args, **kwds)
capitalize()
Return a capitalized version of the string.
casefold()
Return a version of the string suitable for caseless comparisons.
center(width[, fillchar])
Return a centered string of length width.
count(sub[, start[, end]])
Return the number of non-overlapping occurrences of substring sub in string S[start:end].
encode([encoding, errors])
Encode the string using the codec registered for encoding.
endswith(suffix[, start[, end]])
Return True if S ends with the specified suffix, False otherwise.
expandtabs([tabsize])
Return a copy where all tab characters are expanded using spaces.
find(sub[, start[, end]])
Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end].
format(*args, **kwargs)
Return a formatted version of S, using substitutions from args and kwargs.
format_map(mapping)
Return a formatted version of S, using substitutions from mapping.
index(sub[, start[, end]])
Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end].
isalnum()
Return True if the string is an alpha-numeric string, False otherwise.
isalpha()
Return True if the string is an alphabetic string, False otherwise.
isascii()
Return True if all characters in the string are ASCII, False otherwise.
isdecimal()
Return True if the string is a decimal string, False otherwise.
isdigit()


Return True if the string is a decimal string, False otherwise.
isdigit()
Return True if the string is a digit string, False otherwise.
isidentifier()
Return True if the string is a valid Python identifier, False otherwise.
islower()
Return True if the string is a lowercase string, False otherwise.
isnumeric()
Return True if the string is a numeric string, False otherwise.
isprintable()
Return True if the string is printable, False otherwise.
isspace()
Return True if the string is a whitespace string, False otherwise.
istitle()
Return True if the string is a title-cased string, False otherwise.
isupper()
Return True if the string is an uppercase string, False otherwise.
join(iterable, /)
Concatenate any number of strings.
ljust(width[, fillchar])
Return a left-justified string of length width.
lower()
Return a copy of the string converted to lowercase.
lstrip([chars])
Return a copy of the string with leading whitespace removed.
maketrans
Return a translation table usable for str.translate().
partition(sep, /)
Partition the string into three parts using the given separator.
removeprefix(prefix, /)
Return a str with the given prefix string removed if present.
removesuffix(suffix, /)
Return a str with the given suffix string removed if present.
replace(old, new[, count])
Return a copy with all occurrences of substring old replaced by new.
rfind(sub[, start[, end]])
Return the highest index in S where substring sub is found, such that sub is contained within S[start:end].
rindex(sub[, start[, end]])
Return the highest index in S where substring sub is found, such that sub is contained within S[start:end].


rjust(width[, fillchar])
Return a right-justified string of length width.
rpartition(sep, /)
Partition the string into three parts using the given separator.
rsplit([sep, maxsplit])
Return a list of the substrings in the string, using sep as the separator string.
rstrip([chars])
Return a copy of the string with trailing whitespace removed.
split([sep, maxsplit])
Return a list of the substrings in the string, using sep as the separator string.
splitlines([keepends])
Return a list of the lines in the string, breaking at line boundaries.
startswith(prefix[, start[, end]])
Return True if S starts with the specified prefix, False otherwise.
strip([chars])
Return a copy of the string with leading and trailing whitespace removed.
swapcase()
Convert uppercase characters to lowercase and lowercase characters to uppercase.
title()
Return a version of the string where each word is titlecased.
translate(table, /)
Replace each character in the string using the given translation table.
upper()
Return a copy of the string converted to uppercase.
zfill(width, /)
Pad a numeric string with zeros on the left, to fill a field of the given width.
Attributes
QA
Question answering evaluator, which grades answers to questions directly using an LLM.
COT_QA
Chain of thought question answering evaluator, which grades answers to questions using chain of thought 'reasoning'.
CONTEXT_QA
Question answering evaluator that incorporates 'context' in the response.
PAIRWISE_STRING
The pairwise string evaluator, which compares the output of two models.
AGENT_TRAJECTORY
The agent trajectory evaluator, which grades the agent's intermediate steps.
CRITERIA
The criteria evaluator, which evaluates a model based on a custom set of criteria.
STRING_DISTANCE


The criteria evaluator, which evaluates a model based on a custom set of criteria.
STRING_DISTANCE
Compare predictions to a reference answer using string edit distances.
PAIRWISE_STRING_DISTANCE
Compare predictions based on string edit distances.
EMBEDDING_DISTANCE
Compare a prediction to a reference label using embedding distance.
PAIRWISE_EMBEDDING_DISTANCE
Compare two predictions using embedding distance.
capitalize()¶
Return a capitalized version of the string.
More specifically, make the first character have upper case and the rest lower
case.
casefold()¶
Return a version of the string suitable for caseless comparisons.
center(width, fillchar=' ', /)¶
Return a centered string of length width.
Padding is done using the specified fill character (default is a space).
count(sub[, start[, end]]) → int¶
Return the number of non-overlapping occurrences of substring sub in
string S[start:end].  Optional arguments start and end are
interpreted as in slice notation.
encode(encoding='utf-8', errors='strict')¶
Encode the string using the codec registered for encoding.
encodingThe encoding in which to encode the string.
errorsThe error handling scheme to use for encoding errors.
The default is ‘strict’ meaning that encoding errors raise a
UnicodeEncodeError.  Other possible values are ‘ignore’, ‘replace’ and
‘xmlcharrefreplace’ as well as any other name registered with
codecs.register_error that can handle UnicodeEncodeErrors.
endswith(suffix[, start[, end]]) → bool¶
Return True if S ends with the specified suffix, False otherwise.
With optional start, test S beginning at that position.
With optional end, stop comparing S at that position.
suffix can also be a tuple of strings to try.
expandtabs(tabsize=8)¶
Return a copy where all tab characters are expanded using spaces.


Return a copy where all tab characters are expanded using spaces.
If tabsize is not given, a tab size of 8 characters is assumed.
find(sub[, start[, end]]) → int¶
Return the lowest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Return -1 on failure.
format(*args, **kwargs) → str¶
Return a formatted version of S, using substitutions from args and kwargs.
The substitutions are identified by braces (‘{’ and ‘}’).
format_map(mapping) → str¶
Return a formatted version of S, using substitutions from mapping.
The substitutions are identified by braces (‘{’ and ‘}’).
index(sub[, start[, end]]) → int¶
Return the lowest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Raises ValueError when the substring is not found.
isalnum()¶
Return True if the string is an alpha-numeric string, False otherwise.
A string is alpha-numeric if all characters in the string are alpha-numeric and
there is at least one character in the string.
isalpha()¶
Return True if the string is an alphabetic string, False otherwise.
A string is alphabetic if all characters in the string are alphabetic and there
is at least one character in the string.
isascii()¶
Return True if all characters in the string are ASCII, False otherwise.
ASCII characters have code points in the range U+0000-U+007F.
Empty string is ASCII too.
isdecimal()¶
Return True if the string is a decimal string, False otherwise.


isdecimal()¶
Return True if the string is a decimal string, False otherwise.
A string is a decimal string if all characters in the string are decimal and
there is at least one character in the string.
isdigit()¶
Return True if the string is a digit string, False otherwise.
A string is a digit string if all characters in the string are digits and there
is at least one character in the string.
isidentifier()¶
Return True if the string is a valid Python identifier, False otherwise.
Call keyword.iskeyword(s) to test whether string s is a reserved identifier,
such as “def” or “class”.
islower()¶
Return True if the string is a lowercase string, False otherwise.
A string is lowercase if all cased characters in the string are lowercase and
there is at least one cased character in the string.
isnumeric()¶
Return True if the string is a numeric string, False otherwise.
A string is numeric if all characters in the string are numeric and there is at
least one character in the string.
isprintable()¶
Return True if the string is printable, False otherwise.
A string is printable if all of its characters are considered printable in
repr() or if it is empty.
isspace()¶
Return True if the string is a whitespace string, False otherwise.
A string is whitespace if all characters in the string are whitespace and there
is at least one character in the string.
istitle()¶
Return True if the string is a title-cased string, False otherwise.
In a title-cased string, upper- and title-case characters may only
follow uncased characters and lowercase characters only cased ones.
isupper()¶
Return True if the string is an uppercase string, False otherwise.
A string is uppercase if all cased characters in the string are uppercase and


A string is uppercase if all cased characters in the string are uppercase and
there is at least one cased character in the string.
join(iterable, /)¶
Concatenate any number of strings.
The string whose method is called is inserted in between each given string.
The result is returned as a new string.
Example: ‘.’.join([‘ab’, ‘pq’, ‘rs’]) -> ‘ab.pq.rs’
ljust(width, fillchar=' ', /)¶
Return a left-justified string of length width.
Padding is done using the specified fill character (default is a space).
lower()¶
Return a copy of the string converted to lowercase.
lstrip(chars=None, /)¶
Return a copy of the string with leading whitespace removed.
If chars is given and not None, remove characters in chars instead.
static maketrans()¶
Return a translation table usable for str.translate().
If there is only one argument, it must be a dictionary mapping Unicode
ordinals (integers) or characters to Unicode ordinals, strings or None.
Character keys will be then converted to ordinals.
If there are two arguments, they must be strings of equal length, and
in the resulting dictionary, each character in x will be mapped to the
character at the same position in y. If there is a third argument, it
must be a string, whose characters will be mapped to None in the result.
partition(sep, /)¶
Partition the string into three parts using the given separator.
This will search for the separator in the string.  If the separator is found,
returns a 3-tuple containing the part before the separator, the separator
itself, and the part after it.
If the separator is not found, returns a 3-tuple containing the original string
and two empty strings.


and two empty strings.
removeprefix(prefix, /)¶
Return a str with the given prefix string removed if present.
If the string starts with the prefix string, return string[len(prefix):].
Otherwise, return a copy of the original string.
removesuffix(suffix, /)¶
Return a str with the given suffix string removed if present.
If the string ends with the suffix string and that suffix is not empty,
return string[:-len(suffix)]. Otherwise, return a copy of the original
string.
replace(old, new, count=- 1, /)¶
Return a copy with all occurrences of substring old replaced by new.
countMaximum number of occurrences to replace.
-1 (the default value) means replace all occurrences.
If the optional argument count is given, only the first count occurrences are
replaced.
rfind(sub[, start[, end]]) → int¶
Return the highest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Return -1 on failure.
rindex(sub[, start[, end]]) → int¶
Return the highest index in S where substring sub is found,
such that sub is contained within S[start:end].  Optional
arguments start and end are interpreted as in slice notation.
Raises ValueError when the substring is not found.
rjust(width, fillchar=' ', /)¶
Return a right-justified string of length width.
Padding is done using the specified fill character (default is a space).
rpartition(sep, /)¶
Partition the string into three parts using the given separator.
This will search for the separator in the string, starting at the end. If
the separator is found, returns a 3-tuple containing the part before the


the separator is found, returns a 3-tuple containing the part before the
separator, the separator itself, and the part after it.
If the separator is not found, returns a 3-tuple containing two empty strings
and the original string.
rsplit(sep=None, maxsplit=- 1)¶
Return a list of the substrings in the string, using sep as the separator string.
sepThe separator used to split the string.
When set to None (the default value), will split on any whitespace
character (including \n \r \t \f and spaces) and will discard
empty strings from the result.
maxsplitMaximum number of splits (starting from the left).
-1 (the default value) means no limit.
Splitting starts at the end of the string and works to the front.
rstrip(chars=None, /)¶
Return a copy of the string with trailing whitespace removed.
If chars is given and not None, remove characters in chars instead.
split(sep=None, maxsplit=- 1)¶
Return a list of the substrings in the string, using sep as the separator string.
sepThe separator used to split the string.
When set to None (the default value), will split on any whitespace
character (including \n \r \t \f and spaces) and will discard
empty strings from the result.
maxsplitMaximum number of splits (starting from the left).
-1 (the default value) means no limit.
Note, str.split() is mainly useful for data that has been intentionally
delimited.  With natural text that includes punctuation, consider using
the regular expression module.
splitlines(keepends=False)¶
Return a list of the lines in the string, breaking at line boundaries.
Line breaks are not included in the resulting list unless keepends is given and
true.


Line breaks are not included in the resulting list unless keepends is given and
true.
startswith(prefix[, start[, end]]) → bool¶
Return True if S starts with the specified prefix, False otherwise.
With optional start, test S beginning at that position.
With optional end, stop comparing S at that position.
prefix can also be a tuple of strings to try.
strip(chars=None, /)¶
Return a copy of the string with leading and trailing whitespace removed.
If chars is given and not None, remove characters in chars instead.
swapcase()¶
Convert uppercase characters to lowercase and lowercase characters to uppercase.
title()¶
Return a version of the string where each word is titlecased.
More specifically, words start with uppercased characters and all remaining
cased characters have lower case.
translate(table, /)¶
Replace each character in the string using the given translation table.
tableTranslation table, which must be a mapping of Unicode ordinals to
Unicode ordinals, strings, or None.
The table must implement lookup/indexing via __getitem__, for instance a
dictionary or list.  If this operation raises LookupError, the character is
left untouched.  Characters mapped to None are deleted.
upper()¶
Return a copy of the string converted to uppercase.
zfill(width, /)¶
Pad a numeric string with zeros on the left, to fill a field of the given width.
The string is never truncated.
AGENT_TRAJECTORY = 'trajectory'¶
The agent trajectory evaluator, which grades the agent’s intermediate steps.
CONTEXT_QA = 'context_qa'¶
Question answering evaluator that incorporates ‘context’ in the response.
COT_QA = 'cot_qa'¶
Chain of thought question answering evaluator, which grades
answers to questions using
chain of thought ‘reasoning’.


answers to questions using
chain of thought ‘reasoning’.
CRITERIA = 'criteria'¶
The criteria evaluator, which evaluates a model based on a
custom set of criteria.
EMBEDDING_DISTANCE = 'embedding_distance'¶
Compare a prediction to a reference label using embedding distance.
PAIRWISE_EMBEDDING_DISTANCE = 'pairwise_embedding_distance'¶
Compare two predictions using embedding distance.
PAIRWISE_STRING = 'pairwise_string'¶
The pairwise string evaluator, which compares the output of two models.
PAIRWISE_STRING_DISTANCE = 'pairwise_string_distance'¶
Compare predictions based on string edit distances.
QA = 'qa'¶
Question answering evaluator, which grades answers to questions
directly using an LLM.
STRING_DISTANCE = 'string_distance'¶
Compare predictions to a reference answer using string edit distances.


langchain.evaluation.string_distance.base.PairwiseStringDistanceEvalChain¶
class langchain.evaluation.string_distance.base.PairwiseStringDistanceEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, distance: StringDistance = StringDistance.LEVENSHTEIN)[source]¶
Bases: _RapidFuzzChainMixin, PairwiseStringEvaluator
Compute string edit distances between two predictions.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param distance: langchain.evaluation.string_distance.base.StringDistance = StringDistance.LEVENSHTEIN¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶


for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only


addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Asynchronously execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults
to False.
Returns
A dict of named outputs. Should contain all outputs specified inChain.output_keys.
apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶
Call the chain on all inputs in the list.
async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."
dict(**kwargs: Any) → Dict¶
Return dictionary representation of chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
**kwargs – Keyword arguments passed to default pydantic.BaseModel.dict
method.
Returns
A dictionary representation of the chain.
Example
..code-block:: python
chain.dict(exclude_unset=True)
# -> {“_type”: “foo”, “verbose”: False, …}
prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶
Validate and prepare chain inputs, including adding inputs from memory.
Parameters
inputs – Dictionary of raw inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
Returns
A dictionary of all inputs, including those added by the chain’s memory.
prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶


Validate and prepare chain outputs, and save info about this run to memory.
Parameters
inputs – Dictionary of chain inputs, including any inputs added by chain
memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False,
inputs are also added to the final outputs.
Returns
A dict of the final chain outputs.
validator raise_callback_manager_deprecation  »  all fields¶
Raise deprecation warning if callback_manager is used.
run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Convenience method for executing chain when there’s a single string output.
The main difference between this method and Chain.__call__ is that this methodcan only be used for chains that return a single string output. If a Chain
has more outputs, a non-string output, or you want to return the inputs/run
info along with the outputs, use Chain.__call__.
The other difference is that this method expects inputs to be passed directly in
as positional arguments or keyword arguments, whereas Chain.__call__ expects
a single input dictionary with all the inputs.
Parameters
*args – If the chain expects a single input, it can be passed in as the
sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in
addition to tags passed to the chain during construction, but only
these runtime tags will propagate to calls to other objects.


these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in
directly as keyword arguments.
Returns
The chain output as a string.
Example
# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."
# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."
save(file_path: Union[Path, str]) → None¶
Save the chain.
Expects Chain._chain_type property to be implemented and for memory to benull.
Parameters
file_path – Path to file to save the chain to.
Example
chain.save(file_path="path/chain.yaml")
validator set_verbose  »  verbose¶
Set the chain verbosity.
Defaults to the global setting if not specified by the user.
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
validator validate_dependencies  »  all fields¶
Validate that the rapidfuzz library is installed.
Parameters
values (Dict[str, Any]) – The input values.
Returns
The validated values.
Return type
Dict[str, Any]
property evaluation_name: str¶
property input_keys: List[str]¶
Get the input keys.
Returns
The input keys.
Return type
List[str]
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the


property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
property metric: Callable¶
Get the distance metric function.
Returns
The distance metric function.
Return type
Callable
property output_keys: List[str]¶
Get the output keys.
Returns
The output keys.
Return type
List[str]
model Config¶
Bases: object
Configuration for this pydantic object.
arbitrary_types_allowed = True¶


langchain.evaluation.run_evaluators.base.RunEvaluatorOutputParser¶
class langchain.evaluation.run_evaluators.base.RunEvaluatorOutputParser(*, eval_chain_output_key: str = 'text')[source]¶
Bases: BaseOutputParser[EvaluationResult]
Parse the output of a run.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param eval_chain_output_key: str = 'text'¶
dict(**kwargs: Any) → Dict¶
Return dictionary representation of output parser.
get_format_instructions() → str¶
Instructions on how the LLM output should be formatted.
abstract parse(text: str) → T¶
Parse a single string model output into some structure.
Parameters
text – String output of language model.
Returns
Structured output.
parse_chain_output(output: Dict[str, Any]) → EvaluationResult[source]¶
Parse the output of a run.
parse_result(result: List[Generation]) → T¶
Parse a list of candidate model Generations into a specific format.
The return value is parsed from only the first Generation in the result, whichis assumed to be the highest-likelihood Generation.
Parameters
result – A list of Generations to be parsed. The Generations are assumed
to be different candidate outputs for a single model input.
Returns
Structured output.
parse_with_prompt(completion: str, prompt: PromptValue) → Any¶
Parse the output of an LLM call with the input prompt for context.
The prompt is largely provided in the event the OutputParser wants
to retry or fix the output in some way, and needs information from
the prompt to do so.
Parameters
completion – String output of language model.
prompt – Input PromptValue.
Returns
Structured output


prompt – Input PromptValue.
Returns
Structured output
to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶
to_json_not_implemented() → SerializedNotImplemented¶
property lc_attributes: Dict¶
Return a list of attribute names that should be included in the
serialized kwargs. These attributes must be accepted by the
constructor.
property lc_namespace: List[str]¶
Return the namespace of the langchain object.
eg. [“langchain”, “llms”, “openai”]
property lc_secrets: Dict[str, str]¶
Return a map of constructor argument names to secret ids.
eg. {“openai_api_key”: “OPENAI_API_KEY”}
property lc_serializable: bool¶
Return whether or not the class is serializable.
model Config¶
Bases: object
extra = 'ignore'¶


langchain.evaluation.run_evaluators.implementations.get_qa_evaluator¶
langchain.evaluation.run_evaluators.implementations.get_qa_evaluator(llm: BaseLanguageModel, *, prompt: Union[PromptTemplate, str] = PromptTemplate(input_variables=['query', 'result', 'answer'], output_parser=None, partial_variables={}, template="You are a teacher grading a quiz.\nYou are given a question, the student's answer, and the true answer, and are asked to score the student answer as either CORRECT or INCORRECT.\n\nExample Format:\nQUESTION: question here\nSTUDENT ANSWER: student's answer here\nTRUE ANSWER: true answer here\nGRADE: CORRECT or INCORRECT here\n\nGrade the student answers based ONLY on their factual accuracy. Ignore differences in punctuation and phrasing between the student answer and true answer. It is OK if the student answer contains more information than the true answer, as long as it does not contain any conflicting statements. Begin! \n\nQUESTION: {query}\nSTUDENT ANSWER: {result}\nTRUE ANSWER: {answer}\nGRADE:", template_format='f-string', validate_template=True), input_key: str = 'input', prediction_key: str = 'output', answer_key: str = 'output', evaluation_name: Optional[str] = None, **kwargs: Any) → RunEvaluatorChain[source]¶
Get an eval chain that compares response against ground truth.


langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEval¶
class langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEval(score, reasoning)[source]¶
Bases: NamedTuple
Create new instance of TrajectoryEval(score, reasoning)
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
Attributes
reasoning
Alias for field number 1
score
Alias for field number 0
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.
reasoning: str¶
Alias for field number 1
score: int¶
Alias for field number 0


langchain.evaluation.run_evaluators.loading.load_run_evaluator_for_model¶
langchain.evaluation.run_evaluators.loading.load_run_evaluator_for_model(evaluator: EvaluatorType, model: Union[Chain, BaseLanguageModel, Tool], *, input_key: Optional[str] = None, prediction_key: Optional[str] = None, reference_key: Optional[str] = None, eval_llm: Optional[BaseLanguageModel] = None, **kwargs: Any) → List[RunEvaluator][source]¶
Load evaluators specified by a list of evaluator types.
Parameters
evaluator (EvaluatorType) – The evaluator type to load.
model (Union[Chain, BaseLanguageModel, Tool]) – The model to evaluate. Used to infer how to parse the run.
input_key (Optional[str], a chain run's input key to map) – to the evaluator’s input
prediction_key (Optional[str], the key in the run's outputs to) – represent the Chain prediction
reference_key (Optional[str], the key in the dataset example (row)) – outputs to represent the reference, or ground-truth label
eval_llm (BaseLanguageModel, optional) – The language model to use for evaluation, if none is provided, a default
ChatOpenAI gpt-4 model will be used.
**kwargs (Any) – Additional keyword arguments to pass to all evaluators.
Returns
The loaded Run evaluator.
Return type
RunEvaluator


langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain¶
class langchain.evaluation.comparison.eval_chain.PairwiseStringEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, prompt: BasePromptTemplate, llm: BaseLanguageModel, output_key: str = 'text', output_parser: BaseOutputParser = None, return_final_only: bool = True, llm_kwargs: dict = None)[source]¶
Bases: PairwiseStringEvaluator, LLMEvalChain, LLMChain
A chain for comparing two outputs, such as the outputs
of two models, prompts, or outputs of a single model on similar inputs.
Example:
>>> from langchain.chat_models import ChatOpenAI
>>> from langchain.evaluation.comparison import PairwiseStringEvalChain
>>> llm = ChatOpenAI(temperature=0)
>>> chain = PairwiseStringEvalChain.from_llm(llm=llm)
>>> result = chain.evaluate_string_pairs(
…     input = “What is the chemical formula for water?”,
…     prediction = “H2O”,
…     prediction_b = (
…        “The chemical formula for water is H2O, which means”
…        ” there are two hydrogen atoms and one oxygen atom.”
…     referenc = “The chemical formula for water is H2O.”,
… )
>>> print(result[“text”])
# {
#    “value”: “B”,
#    “comment”: “Both responses accurately state”
#       ” that the chemical formula for water is H2O.”


#       ” that the chemical formula for water is H2O.”
#       ” However, Response B provides additional information”
# .     ” by explaining what the formula means.
[[B]]”# }
param callback_manager: Optional[BaseCallbackManager] = None¶
Deprecated, use callbacks instead.
param callbacks: Callbacks = None¶
Optional list of callback handlers (or callback manager). Defaults to None.
Callback handlers are called throughout the lifecycle of a call to a chain,
starting with on_chain_start, ending with on_chain_end or on_chain_error.
Each custom chain can optionally call additional callback methods, see Callback docs
for full details.
param llm: BaseLanguageModel [Required]¶
Language model to call.
param llm_kwargs: dict [Optional]¶
param memory: Optional[BaseMemory] = None¶
Optional memory object. Defaults to None.
Memory is a class that gets called at the start
and at the end of every chain. At the start, memory loads variables and passes
them along in the chain. At the end, it saves any returned variables.
There are many different types of memory - please see memory docs
for the full catalog.
param metadata: Optional[Dict[str, Any]] = None¶
Optional metadata associated with the chain. Defaults to None
This metadata will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param output_parser: BaseOutputParser [Optional]¶
Output parser to use.
Defaults to one that takes the most likely string but does not change it
otherwise.
param prompt: BasePromptTemplate [Required]¶
Prompt object to use.
param return_final_only: bool = True¶


Prompt object to use.
param return_final_only: bool = True¶
Whether to return only the final parsed result. Defaults to True.
If false, will return a bunch of extra information about the generation.
param tags: Optional[List[str]] = None¶
Optional list of tags associated with the chain. Defaults to None
These tags will be associated with each call to this chain,
and passed as arguments to the handlers defined in callbacks.
You can use these to eg identify a specific instance of a chain with its use case.
param verbose: bool [Optional]¶
Whether or not run in verbose mode. In verbose mode, some intermediate logs
will be printed to the console. Defaults to langchain.verbose value.
__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶
Execute the chain.
Parameters
inputs – Dictionary of inputs, or single input if chain expects
only one param. Should contain all inputs specified in
Chain.input_keys except for inputs that will be set by the chain’s
memory.
return_only_outputs – Whether to return only outputs in the
response. If True, only new keys generated by this chain will be
returned. If False, both input keys and new keys generated by this
chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in
addition to callbacks passed to the chain during construction, but only
these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in