Does Character-level Information Always Improve DRS -based Semantic Parsing? (2023)

Abstract: Even in the era of massive language models, it has been suggested that character-level representations improve the performance of neural models. The state-of-the-art neural semantic parser for Discourse Representation Structures uses character-level representations, improving performance in the four languages (i.e., English, German, Dutch, and Italian) in the Parallel Meaning Bank dataset. However, how and why character-level information improves the parser’s performance remains unclear. This study provides an in-depth analysis of performance changes by order of character sequences. In the experiments, we compare F1-scores by shuffling the order and randomizing character sequences after testing the performance of character-level information. Our results indicate that incorporating character-level information does not improve the performance in English and German. In addition, we find that the parser is not sensitive to correct character order in Dutch. Nevertheless, performance improvements are observed when using character-level information.

## introduction character-level information is sometimes helpful in grasping the meanings of words for humans. previous studies have suggested that character-level information helps to improve the performance of neural models on various nlp tasks @xcite . in multilingual nlp systems, character-level information contributes to performance improvements on named entity recognition tasks @xcite and semantic parsing tasks @xcite . however, due to the black-box nature of neural models, it is still unclear how and why character-level information contributes to model performance. the rapid developments of neural models have led to a growing interest in investigating the extent to which these models understand natural language. recent works have indicated that pre-trained language models are insensitive to word order on permuted english datasets on language understanding tasks @xcite @xcite . meanwhile, other works have shown controversial results regarding inductive biases for word order @xcite , especially in different languages @xcite . in this work, we explore the extent to which neural models capture character order. by focusing on character order rather than word order, we present an in-depth analysis of the capacity of models to capture syntactic structures across languages. to analyze whether the importance of character order information differs across languages, we investigate multilingual discourse representation structure (drs; @xcite ) parsing models. @xcite proposed an encoder-decoder drs parsing model incorporating character-level representations. the study concluded that incorporating character-level representations contributes to performance improvements of the model across languages. however, the underlying mechanism remains unclear. we examine the influence of character-level information on drs-based semantic parsing tasks using the state-of-the-art model @xcite . we analyze whether the model is sensitive to the order of character sequences in various units of granularity (i.e., characters, words, and sentences) across the languages. in addition, we investigate whether the amount of information per character-level token affects the model performance. our data will be publicly available at https://github.com/ ynklab/character_order_analysis . sentence brad pitt is an actor. correct order (unigrams) ^^^b r a d ||| ^^^p i t t ||| i s ||| a n ||| a c t o r ||| . uni a a a a a a a a a a a a a a a a a a a a a a a a a shf (word-level) d a r ^^^b ||| t ^^^p t i ||| i s ||| a n ||| o t c a r ||| . shf (sentence-level) c t r r i i . ||| a ||| d a t ||| b p ||| s t ||| ^^^o n a ^^r nd " i c v , t 9 d j : l ' n 6 0 b 0 1 q w ! j w u q bigrams ^^^b br ra ad d||| |||^^^^^^p pi it tt t||| |||i is s||| |||a an n||| |||a ac ct to or r||| |||. 2017 )) is a multilingual corpus annotated with drss. the pmb contains sentences for four languages (english, german, dutch, and italian) with three levels of drs annotation: gold (fully manually checked), silver (partially manually corrected), and bronze (without manual correction). the pmb also provides semantic tags, which are linguistic annotations for producing drss @xcite . neural drs parsing models there have been various attempts to improve the performance of neural drs parsing models, such as by using graph formats @xcite , stack lstms @xcite , and sequence labeling models @xcite . @xcite proposed a sequence-tosequence model with neural encoders and an attention mechanism @xcite . in the study, the number and type of encoders and the type of embeddings of the pre-trained language models, including bert @xcite , were changed to evaluate the model. moreover, linguistic features and character-level representations were added to the model, concluding that character-level representations contribute to the performance improvements in all four languages, compared to using only bert embeddings as input. sensitivity to word order several studies have analyzed whether generic language models understand word order @xcite @xcite . however, these studies have focused on text classification benchmarks, such as glue @xcite , rather than semantic parsing tasks, such as drs parsing. in addition, these studies did not investigate whether models are sensitive to character order. ## experimental setup we explore whether character-level information influences the predictions of the state-of-the-art drs parsing model using character representations @xcite across languages. this section introduces the common experimental setup. ## method we provide multiple methods to reanalyze whether the drs parsing models van noord et al. ( 2020 ) are sensitive to character-level information across languages in a more fine-grained way. first, we reexamine whether character-level information benefits the model in terms of character sequences compared to the setup without an encoder for characters. second, we examine whether the model trained with correct character order predicts correct drss even with incorrect character sequences obtained using techniques such as shuffling. in the above two methods, we prepare models trained with correct character sequences and evaluate the performance when incorrect character order is input to them. third, we explore the capacity of the models to understand character-level information using unigrams or bigrams of characters as character tokens. by using unigrams, we mean one character at a time, and by using bigrams, we mean two characters at a time. ## results and discussion character contribution for models on the other hand, we can see effectiveness in the other languages: dutch and italian. in particular, an f1-score change of more than 50% can be observed in italian. however, values of uni are far lower than ones of no char in dutch and italian. this tendency suggests that providing incorrect character-level information decreases scores critically when incorporating character-level information is effective. models' sensitivity to character order figure 1 shows the micro averaged, maximum, and minimum f1-scores for each type of character-level information: correct, shf-word (word-level shf), shf-sent (sentence-level shf), rnd, and uni (for comparison). in english (figure 1a ) and german (figure 1b ), only minor changes (1%) were observed in the averaged f1-scores for all types of characters. this observation supports less effectiveness of incorporating character-level information for these two languages. we also experimented with the 2-enc + char model without semantic tags in english and obtained similar trends (see appendix d). in dutch (figure 1c ), even though we can see a slight performance decrease from correct to rnd, shuffling the character order does not affect the performance of the models. these results indicate that drs parsing models are not sensitive to character order for dutch. for italian (figure 1d ), we can see that the correct character order contributes to the performance of the model. shuffling the characters within each word decreased the model's performance by 20% (from 79% to 59%). the performance decreased by another 20% (from 59% to 39%) when shuf- extending character tokens improves model performance one of the reasons models cannot achieve any improvements in english, while improvements are observed in non-english languages, is the quantity and quality of data in the pmb. as noted in the statistics of pmb 3.0.0 (appendix b and ## conclusion and future work in this study, we carried out a further exploration of the extent to which character-level representations contribute to the performance improvements of multilingual drs parsing models. we found that character-level information provided little performance improvement in english and german but improved performance in dutch and italian. however, we find that the model is sensitive to character order in italian but not in dutch. the take-away message from our investigation is that the importance of character-level information in drs-based semantic parsing depends on the language and syntactic structures of the sentences. in future work, we will analyze in more detail the significant differences between the four languages, especially italian, and other languages. another direction of our future work is to investigate the relationship between the neural models and humans in reading performance for incorrect character order. it would be interesting to analyze whether the results on drs parsing tasks are consistent with those of these studies @xcite @xcite .