We propose USIM, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USIM operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USIM, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USIM similarity score to the source; and (3) invalid corrections obtain a lower score. 1
The USIM metric REF measures the semantic faithfulness of the output to the source by comparing their respective UCCA graphs.
Reference-less Measure of Faithfulness for Grammatical Error Correction
ArXiv
ArXiv
Computer Science