A distributed system is described that reliably mines parallel text from large corpora. The approach can be regarded as cross-language near-duplicate detection, enabled by an initial, low-quality batch translation. In contrast to other approaches which require specialized metadata, the system uses only the textual content of the documents. Results are presented for a corpus of over two billion web pages and for a large collection of digitized public-domain books.
REF proposed a large scale distributed system to mine parallel text from the web and books.
Large Scale Parallel Document Mining for Machine Translation
COLING
Computer Science