Une note sur l’analyse du constituant pour le français (A Note on constituent parsing for F rench) (2018)

Abstract: Cet article traite des analyses d’erreurs quantitatives et qualitatives sur les résultats de l’analyse syntaxique des constituants pour le français. Pour cela, nous étendons l’approche de Kummerfeld et al. (2012) pour français, et nous présentons les détails de l’analyse. Nous entraînons les systèmes d’analyse syntaxique statistiques et neuraux avec le corpus arboré pour français, et nous évaluons les résultats d’analyse. Le corpus arboré pour le français fournit des étiquettes syntagmatiques à grain fin, et les caractéristiques grammaticales du corpus affectent des erreurs d’analyse syntaxique.

## experiments and results the current available version of the french treebank contains 45 files and 21,550 sentences @xcite . 1 we use a corpus split proposed in @xcite for training, development and test datasets directly from the french treebank instead of the distribution version from the spmrl 2013 shared task. 2 this is mainly to train/evaluate the treebank using the different annotation such as training with functional information. while there are more sentences in the current treebank with 17,774/1,235/2,541 sentences for training/dev/evaluation, we use the exact data split from @xcite (14,759/1,235/2,541). for statistical parsing using the berkeley parser (petrov et al., 2006) 3 , we report evaluation results using grammars which give the best results on development data. while the original berkeley parser proposed several runs of training because of the em algorithm which can find locally maximum likelihood parameters, we empirically found that each run of training gives same results. therefore, we use the single run of training using the berkeley parser with the default option. for experiments in this paper, we use penn treebank-like preprocessing, especially by removing null elements ( * t * ) and functional information in the phrase label (e.g. -suj or -obj) as described in @xcite . we evaluate the parser accuracy with the standard f 1 metric from evalb. 4 while the spmrl shared task provides the alternative evalb 5 , it produces the same f 1 scores for french. we only change the original evalb to display results for @xmath0. we rename phrase labels which share the same label names with pos labels (usually for multi-word expressions or compound words) (+r). for example, we convert [ p [ p d'][ p après]] into [ p+ [ p d'][ p après]] to differentiate between ps in the phrase label and the pos label. therefore, we rename pos labels a, adv, c, cl, d, et, i, n, p, pro, and v which also appear in the phrase labels. we note that the treebank of the spmrl shared task has a similar annotation for compound words. for comparison reason, we also use functional information during training (+f) without renaming phrase labels. for example np+suj and vppart+mod instead of np and vppart are used for (+f). for neural parsing, we use the trance parser @xcite 6 and a pre-trained 300 dimension embedding vector provided by @xcite 7 . we use default options with 50 epochs for the trance parser. ## parsing error analysis recent state of the art parsing techniques are easily trained and evaluated if the syntactically annotated treebank is available. their results, however, can be difficult to understand because grammars are automatically induced from the treebank. @xcite presented an approach to quantify constituent parsing errors based on the treebank annotation. 8 in this section, we extend kummerfeld's approach to the french treebank parsing for parser error analysis. error analysis is based on parsing results (+r). co-ordination error annotating phrase with co-ordination in french is a difficult problem (inter alia mouret ( 2007 )). the current annotation in the french treebank shows a hierarchical structure, which is different with the english penn treebank (a flat structure). finding the correct scope of the coordinating conjunction is challenging, and co-ordination errors occur frequently. see figure 2 for an example of the co-ordination error. different label a phrase label is wrongly assigned. we note that pos label errors are not counted, and even for parsing with gold pos label, the berkeley parser does not always obtain 100% for pos labeling accuracy. see figure 3 for an example of the different label error. np internal structure a general structure of the french treebank is relatively flat for the inside of np as well as the entire sentence. for example, a sentence in ( 4 ) is an np with a flat structure as follows : however, both parsers fail to capture the flat structure for np including a phrase segmentation. see figure 4 for an example of the np internal structure error. we do not detail single word and unary errors because they are mostly parts of another errors. over 30% of parsing errors are undefined. we need to investigate these other error types for constituent parsing results, which can be more pertinent for french. we leave this for future work. previously, @xcite proposed an error mining technique based on parsing results from the frmg @xcite and the sxlfg @xcite parsers. since they parsed the raw corpus without knowing the correct parsed tree, they tried to find "suspicious" parsed results. these suspicious parsing trees are calculated based on predefined syntactic-related resources such as a morphological and syntactic lexicon lefff @xcite and a pre-syntactic processing pipeline sxpipe @xcite . ## discussion and conclusion this paper dealt with error analysis studies on french constituent parsing results. while a neural parser improved parsing results for other languages such as english and chinese, we did not obtain the better results for french. there are many intrinsic (learning rate, dropout, # of epochs, etc.) and extrinsic (word embedding and its dimension size) factors. since training using 50 epochs takes over three or four days to learn a parsing model on a single machine, it wouldn't be easy to find proper parameters for french for neural parsing. we leave finding optimal parameter for french to future work. functional information would also improve parsing results for certain morphologically rich languages @xcite . the french treebank provides fine-grained phrase labels (111 different labels) and the berkeley parser also generates additional internal phrase labels during training. pcfg rules in berkeley+f contain over 2m, compared to 0.4m in berkeley+r (cf. 18k vs. 15k for trance+f and trance+r). such diversities with phrase labels can give a biased distribution. therefore, functional information hardly effects or even tends to worsen parsing results in many cases. investigating the effective way on clustering phrase labels can be one direction to improve parsing results and we leave this for future work. instead of renaming phrase labels with +, we can also consider renaming with existing * p-like labels such as np or pp : e.g. the phrase label of compound words in [ np [ n [ n banques] [ a centrales]]] is converted into [ np [ np [ n ... ] [ a ... ] ]] instead of [ np [ n+ [ n ... ] [ a ... ]]]. using * p-like renaming labels (12 phrase labels), we can find additional repetitive unary branches and we remove them during preprocessing. converting into * p-like labels is straightforward except for d in which it would be np for numbers such as [ np [ d vingt] [ ponct -] [ d cinq] [ d mille]] ; otherwise, pp. consequently, we have 284,107 non-terminal nodes instead of 288,374 as in berkeley+r (excluding pre-terminal pos labels) in training data, and obtain only up to 75.42% f 1 score. this reflects the fact that recognizing multi-word expressions (mwes) and compound words is important in parsing for french, and it already proved in le roux et al. (2014) where they employed external linguistic resources such as delac, compound word dictionary for french (courtois et al., 1997) 9 . exploring mwes can be another direction to improve parsing results. 10 we note that we obtained slightly better results using the berkeley parser than what the spmrl shared task reported (gold setting) : 80.38 and 81.76 for w/o and w/ gold pos labels. this is probably because a preprocessing step for treebank data could be "slightly" dissimilar. we used our own the preprocessed french treebank to explore the different treebank settings.