Word segmentation in MSR-NLP is an integral part of a sentence analyzer which includes basic segmentation, derivational morphology, named entity recognition, new word identification, word lattice pruning and parsing. The final segmentation is produced from the leaves of parse trees. The output can be customized to meet different segmentation standards through the value combinations of a set of parameters. The system participated in four tracks of the segmentation bakeoff --PK-open, PK-close, CTB-open and CTBclosed -and ranked #1, #2, #2 and #3 respectively in those tracks. Analysis of the results shows that each component of the system contributed to the scores. The MSR-NLP Chinese system that participated in the current segmentation bakeoff is not a standalone word segmenter. It is a Chinese sentence analyzer where the leaves of parse trees are displayed as the output of word segmentation. The components of this system are described below. Each input sentence is first segmented into individual characters. 1 These characters and their combinations are then looked up in a dictionary 2 and a word lattice containing lexicalized words only is formed. Each node in the lattice is a feature 1 If an input line contains more than one sentence, a sentence separator is applied to break the line into individual sentences, which are then processed one by one and the results are concatenated to form a single output. 2 The lookup is optimized so that not all possible combinations are tried. matrix that contains the part of speech and other grammatical attributes. Multiple-character words may also have information for resolving segmentation ambiguities. In general, multiple-character words are assigned higher scores than the words they subsume, but words like "才能" are exceptions and such exceptional cases are usually marked in the dictionary. For some of the words that tend to overlap with other words, there is also information as to what the preferred segmentation is. For instance, the preferred segmentation for "会议员" is "会+议员" rather than "会议+员". Such information was collected from segmented corpora and stored in the dictionary. The scores are later used in word lattice pruning and parse ranking (Wu and Jiang 1998). After basic segmentation, a set of augmented phrase structure rules are applied to the word lattice to form larger word units which include: • Words derived from morphological processes such as reduplication, affixation, compounding, merging, splitting, etc. • Named entities such as person names, place names, company names, product names, numbers, dates, monetary units, etc. Each of these units is a tree that reflects the history of rule application. They are added to the existing word lattice as single nodes and treated as single words by the parser. The internal structures are useful for various purposes, one of which is the customization of word segmentation: words with such structures can all be displayed as single words or multiple words depending on where the "cuts" are made in the word tree (Wu 2003) .
REF also used the training data to tune the segmentation parameters of their MSR-NLP Chinese system.
Chinese Word Segmentation In MSR-NLP
SIGHAN Workshop On Chinese Language Processing
Computer Science