jordimas/gec-opennmt-english

Introduction

This repository contains a description on how to use OpenNMT on the Grammar Error Correction (GEC) task. The idea is to approch GEC as a translation task

Usage

Install the necessary dependencies:

pip3 install ctranslate2 pyonmttok

Simple tokenization & translation using Python:

import ctranslate2
import pyonmttok
from huggingface_hub import snapshot_download
model_dir = snapshot_download(repo_id="jordimas/gec-opennmt-english", revision="main")

tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
tokenized=tokenizer.tokenize("The water are hot. My friends are going to be late. Today mine mother is in Barcelona.")

translator = ctranslate2.Translator(model_dir)
translated = translator.translate_batch([tokenized[0]])
print(tokenizer.detokenize(translated[0][0]['tokens']))

Model

The model has been training using the clang8 corpus for English language.

Details:

• Model: TransformerBase
• Tokenizer: SentencePiece
• BLEU = 85.50

Papers

Relevant papers:

• Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task
• A Simple Recipe for Multilingual Grammatical Error Correction

Contact

Email address: Jordi Mas: jmas@softcatala.org