the evaluation of legal knowledge based systems
Evaluation strategies to assess the effectiveness of legal knowledge based systems enable strengths and limitations of systems to be accurately articulated. This facilitates efforts in the research community to develop systems and also promotes the adoption of research prototypes in the commercial world. However, evaluation strategies for systems that operate in a domain as complex as law are difficult to specify. In this paper, we present an evaluation framework put forward by Reich and describe how this motivated the evaluation of our systems in Australian family law. Strategies surveyed include a comparison of linear regression with neural networks, user acceptance surveys, a comparison of system predictions with those from past cases, and a comparison of system outputs with those proposed by a panel of lawyers. Specific criteria for the evaluation of explanation facilities are also described.
evaluation legal knowledge based system