Generating Phonetic Embeddings for B ulgarian Words with Neural Networks (2024)

Abstract: Word embeddings can be considered the cornerstone of modern natural language processing. They are used in many NLP tasks and allow us to create models that can understand the meaning of words. Most word embeddings model the semantics of the words. In this paper, we create phoneme-based word embeddings, which model how a word sounds. This is accomplished by training a neural network that can automatically generate transcriptions of Bulgarian words. We used the Jaccard index and direct comparison metrics to measure the performance of neural networks. The models perform nearly perfectly with the task of generating transcriptions. The model’s word embeddings offer versatility across various applications, with its application in automatic paronym detection being particularly notable, as well as the task of detecting the language of origin of a Bulgarian word. The performance of this paronym detection is measured with the standard classifier metrics - accuracy, precision, recall, and F1.

## introduction in natural language processing, automatically generating transcriptions of words is a formidable challenge, particularly when confronted with languages like bulgarian. renowned for its nearperfect phonemic orthography, wherein each letter typically corresponds to a single sound, bulgarian presents an intriguing paradox. while its orthographic structure promises clarity, exceptions within the system thwart the straightforward application of bijective mapping algorithms. these anomalies underscore the need for innovative approaches that seamlessly reconcile orthographic and phonetic representations. in this work, we create a neural network that generates a phoneme transcription from a word. the model is trained on five books, transforming each into a pair of words and transcriptions. one particularly noteworthy application of our model lies in automatic paronym detection. by seamlessly integrating phonetic representations into the detection process, our model demonstrates good results for paronym detection. another application of phoneme embeddings is the detection of loanwords in the bulgarian language. a classifier is trained using the embeddings as input, which classifies the language of origin of a given word. ## related work generating the phoneme transcriptions from the grapheme representation is not new, and multiple approaches have been proposed. @xcite and @xcite both describe a rule-based approach where graphemes are directly replaced with their corresponding phonemes. on an english language dataset, @xcite 's approach achieves around 80% of correctly generated transcriptions, while @xcite 's approach results in around 70% of correctly generated transcriptions. a statistical model is described in @xcite , which relies on modeling the translation process as a linear sequence of operations. the model generates a phonemic transcription from the orthographic form of a word. this is achieved by approximating the sum of the joint probabilities of all possible graphone sequences that match the given spelling. @xcite describes an lstm-based model for performing grapheme-to-phoneme conversion with a 25.8% error rate on a standard english dataset. @xcite describes multiple approaches that work on a single language -a joint n-gram model, a sequence-to-sequence lstm model, and a transformer model. the models handle unknown languages by finding the closest language based on the language family tree and using the nearest k language models. the downside of this approach would be that if a model is not trained on bulgarian data, it might not yield sufficient results. yolchuyeva et al. (2019) and engelhart et al. (2021) propose using transformer-based models to automatically generate the transcriptions from words and apply the model to different tasks for the english language. in this paper, a similar transformer-based network to @xcite and @xcite will be used, with the main difference being that the network will be trained exclusively on bulgarian words and transcriptions to solve the task of automatic transcription generation for the bulgarian language. ## usage of the phonetic embeddings in this section, we will use the following notation: the embeddings result from using only the encoder part of the model. the user enters a word w , which is transformed (4.1) into the input vector for the model v i w . from there, the vector is passed to the encoder part of the model, which returns the vector e w . as the model works on syllables, the returned embedding is not a single vector, but l w vectors, each of size emb (which in this case is 512), containing floating-point numbers. ## experiments apart from the notation used in section 4.1, we will introduce the following symbols: ## limitations while the results of the model are looking good, there are some limitations on it. the input of the model does not include information about the emphasis of the word. this information is only used to generate the proper transcription for the training data. this impacts the phonemes generated by the model. the phonetic embeddings also don't seem to exhibit any of the properties present in other word embeddings. embeddings like word2vec @xcite represent the semantic and syntactic relationships between the words. for example, the distance between the words "man" and "woman" is similar to the distance between the words "king" and "queen". the phonetic embeddings however don't exhibit such connections. for example, if we have two words, which differ only at the suffix, the distance between their embeddings varies from small to a large, and is not consistent through different pairs of words. ## conclusion a dataset containing 600,000+ words and their transcription was created. a transformer-based model was created to solve the bulgarian language's grapheme-to-phoneme task. the model performs with very high accuracy. embeddings extracted from this model were used in a simple classifier that checks if two words are paronyms. the classifier performs also with a high accuracy percentage. as a result, there is now an automatic system for paronym detection and automatic generation of phonemic transcriptions of bulgarian words. these approaches can be applied to other languages that are different from bulgarian. in future research, we intend to use the embeddings to detect if a word is a loanword.