## JSON Evaluators

Evaluating [extraction](https://python.langchain.com/docs/use_cases/extraction/) and function calling applications often comes down to validation that the LLM’s string output can be parsed correctly and how it compares to a reference object. The following `JSON` validators provide functionality to check your model’s output consistently.

## JsonValidityEvaluator[​](#jsonvalidityevaluator "Direct link to JsonValidityEvaluator")

The `JsonValidityEvaluator` is designed to check the validity of a `JSON` string prediction.

### Overview:[​](#overview "Direct link to Overview:")

*   **Requires Input?**: No
*   **Requires Reference?**: No

```
from langchain.evaluation import JsonValidityEvaluatorevaluator = JsonValidityEvaluator()# Equivalently# evaluator = load_evaluator("json_validity")prediction = '{"name": "John", "age": 30, "city": "New York"}'result = evaluator.evaluate_strings(prediction=prediction)print(result)
```

```
prediction = '{"name": "John", "age": 30, "city": "New York",}'result = evaluator.evaluate_strings(prediction=prediction)print(result)
```

```
{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes: line 1 column 48 (char 47)'}
```

## JsonEqualityEvaluator[​](#jsonequalityevaluator "Direct link to JsonEqualityEvaluator")

The `JsonEqualityEvaluator` assesses whether a JSON prediction matches a given reference after both are parsed.

### Overview:[​](#overview-1 "Direct link to Overview:")

*   **Requires Input?**: No
*   **Requires Reference?**: Yes

```
from langchain.evaluation import JsonEqualityEvaluatorevaluator = JsonEqualityEvaluator()# Equivalently# evaluator = load_evaluator("json_equality")result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 1}')print(result)
```

```
result = evaluator.evaluate_strings(prediction='{"a": 1}', reference='{"a": 2}')print(result)
```

The evaluator also by default lets you provide a dictionary directly

```
result = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})print(result)
```

## JsonEditDistanceEvaluator[​](#jsoneditdistanceevaluator "Direct link to JsonEditDistanceEvaluator")

The `JsonEditDistanceEvaluator` computes a normalized Damerau-Levenshtein distance between two “canonicalized” JSON strings.

### Overview:[​](#overview-2 "Direct link to Overview:")

*   **Requires Input?**: No
*   **Requires Reference?**: Yes
*   **Distance Function**: Damerau-Levenshtein (by default)

_Note: Ensure that `rapidfuzz` is installed or provide an alternative `string_distance` function to avoid an ImportError._

```
from langchain.evaluation import JsonEditDistanceEvaluatorevaluator = JsonEditDistanceEvaluator()# Equivalently# evaluator = load_evaluator("json_edit_distance")result = evaluator.evaluate_strings(    prediction='{"a": 1, "b": 2}', reference='{"a": 1, "b": 3}')print(result)
```

```
{'score': 0.07692307692307693}
```

```
# The values are canonicalized prior to comparisonresult = evaluator.evaluate_strings(    prediction="""    {        "b": 3,        "a":   1    }""",    reference='{"a": 1, "b": 3}',)print(result)
```

```
# Lists maintain their order, howeverresult = evaluator.evaluate_strings(    prediction='{"a": [1, 2]}', reference='{"a": [2, 1]}')print(result)
```

```
{'score': 0.18181818181818182}
```

```
# You can also pass in objects directlyresult = evaluator.evaluate_strings(prediction={"a": 1}, reference={"a": 2})print(result)
```

```
{'score': 0.14285714285714285}
```

## JsonSchemaEvaluator[​](#jsonschemaevaluator "Direct link to JsonSchemaEvaluator")

The `JsonSchemaEvaluator` validates a JSON prediction against a provided JSON schema. If the prediction conforms to the schema, it returns a score of True (indicating no errors). Otherwise, it returns a score of 0 (indicating an error).

### Overview:[​](#overview-3 "Direct link to Overview:")

*   **Requires Input?**: Yes
*   **Requires Reference?**: Yes (A JSON schema)
*   **Score**: True (No errors) or False (Error occurred)

```
from langchain.evaluation import JsonSchemaEvaluatorevaluator = JsonSchemaEvaluator()# Equivalently# evaluator = load_evaluator("json_schema_validation")result = evaluator.evaluate_strings(    prediction='{"name": "John", "age": 30}',    reference={        "type": "object",        "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},    },)print(result)
```

```
result = evaluator.evaluate_strings(    prediction='{"name": "John", "age": 30}',    reference='{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}',)print(result)
```

```
result = evaluator.evaluate_strings(    prediction='{"name": "John", "age": 30}',    reference='{"type": "object", "properties": {"name": {"type": "string"},'    '"age": {"type": "integer", "minimum": 66}}}',)print(result)
```

```
{'score': False, 'reasoning': "<ValidationError: '30 is less than the minimum of 66'>"}
```

* * *

#### Help us out by providing feedback on this documentation page: