Development of Hybrid Algorithm for Automatic Extraction of Multiword Expressions from Monolingual and Parallel Corpus of E nglish and P unjabi (2020)

Abstract: Identification and extraction of Multiword Expressions (MWEs) is very hard and challenging task in various Natural Language processing applications like Information Retrieval (IR), Information Extraction (IE), Question-Answering systems, Speech Recognition and Synthesis, Text Summarization and Machine Translation (MT). Multiword Expressions are two or more consecutive words but treated as a single word and actual meaning this expression cannot be extracted from meaning of individual word. If any systems recognized this expression as separate words, then results of system will be incorrect. Therefore it is mandatory to identify these expressions to improve the result of the system. In this report, our main focus is to develop an automated tool to extract Multiword Expressions from monolingual and parallel corpus of English and Punjabi. In this tool, Rule based approach, Linguistic approach, statistical approach, and many more approaches were used to identify and extract MWEs from monolingual and parallel corpus of English and Punjabi and achieved more than 90% f-score value in some types of MWEs.

## introduction in this tool, ruled based, linguistic and statistical approaches are used to extract mwes. apart from these approaches, part of speech tagger tool, named entities recognizer tool, stemmer, giza++, etc tools are used. most of mwes are generic in nature, it means based on rules and linguistic approach than statistical approach. mostly ruled based approach is used in replicated words, which are strong candidate of mwes in punjabi language. the results of linguistic approach in english are better than results of punjabi, because of the poor performance of punjabi part of speech tagger tool and punjabi stemmer. similarly results of monolingual corpus are better than parallel corpus, because of the lack of proper punjabi-english dictionary and poor performance of giza++ tool. therefore results of mwes are directly depending upon performance of above mentioned tools. as we earlier discussed, most of the nlp applications are highly affected by mwes. this automatic mwes tool will help the performance of many nlp applications like information retrieval (ir), information extraction (ie), question-answering systems, speech recognition and synthesis, text summarization and machine translation (mt). non-compositional, non-modifiable and nonsubstitutable are basic features of mwes. non compositional means that meaning of mwes cannot be predicted from meaning its parts. non modifiable means that multiword expressions are frozen and they cannot be changed in any way. non substitutable means that any parts of multiword expression cannot be substituted by one of its synonym without affecting the meaning of an expression. @xcite described that noncompositional, non-modifiable and nonsubstitutable are basic features of mwe. ## review of literature the concept of multiword expression is given by @xcite has covered all types of mwes. identification and extraction of mwes is not very old field, but still many of the researchers are working in this field. there has been very limited work done reported on monolingual punjabi mwes and extraction of parallel mwes from english-punjabi parallel corpus. most of researcher used statistical method or association measures tools @xcite , linguistic based approaches @xcite , ruled based approaches (m nandi, 2013), hybrid approaches @xcite for extracting mwes expression from monolingual and parallel corpus. in indian language, @xcite @xcite classified reduplicate mwes which are strong candidates of mwes. 3 methodology this report presents an automatic tool which extracts following multiword expressions from english and punjabi language.