  In a legal system, judgment consistency is regarded as one of the most
important manifestations of fairness. However, due to the complexity of factual
elements that impact sentencing in real-world scenarios, few works have been
done on quantitatively measuring judgment consistency towards real-world data.
In this paper, we propose an evaluation metric for judgment inconsistency,
Legal Inconsistency Coefficient (LInCo), which aims to evaluate inconsistency
between data groups divided by specific features (e.g., gender, region, race).
We propose to simulate judges from different groups with legal judgment
prediction (LJP) models and measure the judicial inconsistency with the
disagreement of the judgment results given by LJP models trained on different
groups. Experimental results on the synthetic data verify the effectiveness of
LInCo. We further employ LInCo to explore the inconsistency in real cases and
come to the following observations: (1) Both regional and gender inconsistency
exist in the legal system, but gender inconsistency is much less than regional
inconsistency; (2) The level of regional inconsistency varies little across
different time periods; (3) In general, judicial inconsistency is negatively
correlated with the severity of the criminal charges. Besides, we use LInCo to
evaluate the performance of several de-bias methods, such as adversarial
learning, and find that these mechanisms can effectively help LJP models to
avoid suffering from data bias.
