In this paper we describe the Helsinki language identification method, HeLI, and the resources we created for and used in the 3rd edition of the Discriminating between Similar Languages (DSL) shared task, which was organized as part of the VarDial 2016 workshop. The shared task comprised of a total of 8 tracks, of which we participated in 7. The shared task had a record number of participants, with 17 teams providing results for the closed track of the test set A. Our system reached the 2nd position in 4 tracks (A closed and open, B1 open and B2 open) and in this paper we are focusing on the methods and data used for those tracks. We describe our wordbased backoff method in mathematical notation. We also describe how we selected the corpus we used in the open tracks.
The complete description of the method was first presented in the proceedings of the 3 rd VarDial workshop REF .
HeLI, a Word-Based Backoff Method for Language Identification
VarDial@COLING
Computer Science