Recurrent neural network-based machine translation models suffer from unbalanced outputs, meaning quality degrades as they generate outputs.

They propose to apply a generic hyperparameter-free agreement model for neural machine translation which can be used in any neural models.

The neural machine translation models with the agreement model generate better balanced outputs and significantly outperform baseline models.

Abstract: Neural machine translation ( NMT ) with recurrent neural networks , has proven to be an effective technique for end-to-end machine translation .

Abstract: However , in spite of its promising advances over traditional translation methods , it typically suffers from an issue of unbalanced outputs , that arise from both the nature of recurrent neural networks themselves , and the challenges inherent in machine translation .

Abstract: To overcome this issue , we propose an agreement model for neural machine translation and show its effectiveness on large-scale Japaneseto-English and Chinese-to-English translation tasks .

Abstract: Our results show the model can achieve improvements of up to 1.4 BLEU over the strongest baseline NMT system .

Abstract: With the help of an ensemble technique , this new end-to-end NMT approach finally outperformed phrasebased and hierarchical phrase-based Moses baselines by up to 5.6 BLEU points .

Recurrent neural network ( RNN ) has achieved great successes on several structured prediction tasks ( Graves , 2013 ; Watanabe and Sumita , 2015 ; Dyer et al . , 2015 ) , in which RNNs are required to make a sequence of dependent predictions .

One of its advantages is that an unbounded history is available to enrich the context for the prediction at the current time-step .

Despite its successes , recently , ( Liu et al . , 2016 ) pointed out that the RNN suffers from a fundamental issue of generating unbalanced outputs : that is to say the suffixes of its outputs are typically worse than the prefixes .

This is due to the fact that later predictions directly depend on the accuracy of previous predictions .

They empirically demonstrated this issue on two simple sequence-to-sequence learning tasks : machine transliteration and grapheme-to-phoneme conversion .

On the more general sequence-to-sequence learning task of machine translation ( MT ) , neural machine translation ( NMT ) based on RNNs has recently become an active research topic ( Sutskever et al . , 2014 ; Bahdanau et al . , 2014 ) .

Compared to those two simple tasks , MT involves in much larger vocabulary and frequent reordering between input and output sequences .

This makes the prediction at each time-step far more challenging .

In addition , sequences in MT are much longer , with averaged length of 36.7 being about 5 times longer than that in grapheme-to-phoneme conversion .

Therefore , we believe that the history is more likely to contain incorrect predictions and the issue of unbalanced outputs may be more serious .

This hypothesis is supported later ( see Table 1 in § 4.1 ) , by an analysis that shows the quality of the prefixes of translation hypotheses is much higher than that of the suffixes .

To address this issue for NMT , in this paper we extend the agreement model proposed in ( Liu et al . , 2016 ) to the task of machine translation .

Its key idea is to encourage the agreement between a pair of target-directional ( left-to-right and right-to-left ) NMT models in order to produce more balanced translations and thus improve the overall translation quality .

• We introduce a simple and general method to address the issue of unbalanced outputs for NMT ( § 3 ) .

This method is robust without any extra hyperparameters to tune and is easy to implement .

In addition , it is general enough to be applied on top of any of the existing RNN translation models , although it was implemented on top of the model in ( Bahdanau et al . , 2014 ) in this paper .

• We provide an empirical evaluation of the technique on large scale Japanese-to-English and Chinese-to-English translation tasks .

The results show our model can generate more balanced translation results , and achieves substantial improvements ( of up to 1.4 BLEU points ) over the strongest NMT baseline ( § 4 ) .

With the help of an ensemble technique , our new end-to-end NMT gains up to 5.6 BLEU points over phrase-based and hierarchical phrasebased Moses ( Koehn et al . , 2007 )

2 Overview of Neural Machine Translation

Suppose x = x 1 , x 2 , • • • , x m denotes a source sentence , y = y 1 , y 2 , • • • , y n denotes a target sen- tence .

In addition , let x < t = x 1 , x 2 , • • • , x t-1 denote a prefix of x.

Neural Machine Translation ( NMT ) directly maps a source sentence into a target within a probabilistic framework .

Formally , it defines a conditional probability over a pair of sequences x and y via a recurrent neural network as follows :

where θ is the set of model parameters ; h t denotes a hidden state ( i.e. a vector ) of y at timestep t ; g is a transformation function from a hidden state to a vector with dimension of the target-side vocabulary size ; softmax is the softmax function , and [ i ] denotes the i th component in a vector .

) is defined by a recurrent function over both the previous hidden state h t-1 and the context c(x , y < t ) .3

Note that both h t and c(x , y < t ) have dimension d for all t.

In this paper , we develop our model on top of the neural machine translation approach of ( Bahdanau et al . , 2014 ) , and we refer the reader this paper for a complete description of the model , for example , the definitons of f and c. The proposed method could just as easily been implemented on top of any other RNN models such as that in ( Sutskever et al . , 2014 ) .

In this paper , we investigate the issue of unbalanced outputs suffered by recurrent neural networks , and empirically show its existence in the context of machine translation .

To address this issue , we propose an easy to implement agreement model that extends the method of ( Liu et al . , 2016 ) from simple sequence-to-sequence learning tasks to machine translation .

On two challenging JP-EN and CH-EN translation tasks , our approach was empirically shown to be effective in addressing the issue ; by generating balanced outputs , it was able to consistently outperform a respectable NMT baseline on all test sets , delivering gains of up to 1.4 BLEU points .

To put these results in the broader context of machine translation research , our approach ( even without special handling of unknown words ) achieved gains of up to 5.6 BLEU points over strong phrase-based and hierarchical phrase-based Moses baselines , with the help of an ensemble technique .


Even with the recent successes of classification models, their applications to information retrieval have been abandoned.

They propose to use a machine learning-based discriminative model to obtain richer features further can be used for information retrieval.

They show that the proposed approach can reach the performance of well-studied heuristics such as TF-IDF and generalizes well to unseen queries.

Abstract: We approached the problem as learning how to order documents by estimated relevance with respect to a user query .

Abstract: Our support vector machines based classifier learns from the relevance judgments available with the standard test collections and generalizes to new , previously unseen queries .

Abstract: For this , we have designed a representation scheme , which is based on the discrete representation of the local ( lw ) and global ( gw ) weighting functions , thus is capable of reproducing and enhancing the properties of such popular ranking functions as tf.idf , BM25 or those based on language models .

Abstract: Our tests with the standard test collections have demonstrated the capability of our approach to achieve the performance of the best known scoring functions solely from the labeled examples and without taking advantage of knowing those functions or their important properties or parameters .

Our work is motivated by the objective to bring closer numerous achievements in the domains of machine learning and classification to the classical task of ad-hoc information retrieval ( IR ) , which is ordering documents by the estimated degree of relevance to a given query .

Although used with striking success for text categorization , classification-based approaches ( e.g. those based on support vector machines , Joachims , 2001 ) have been relatively abandoned when trying to improve ad hoc retrieval in favor of empirical ( e.g. vector space , Salton & McGill , 1983 ) or generative ( e.g. language models ; Zhai & Lafferty 2001 ; Song & Croft ; 1999 ) , which produce a ranking function that gives each document a score , rather than trying to learn a classifier that would help to discriminate between relevant and irrelevant documents and order them accordingly .

A generative model needs to make assumptions that the query and document words are sampled from the same underlying distributions and that the distributions have certain forms , which entail specific smoothing techniques ( e.g. popular Dirichlet-prior ) .

A discriminative ( classifier-based ) model , on the other side , does not need to make any assumptions about the forms of the underlying distributions or the criteria for the relevance but instead , learns to predict to which class a certain pattern ( document ) belongs to based on the labeled training examples .

Thus , an important advantage of a discriminative approach for the information retrieval task , is its ability to explicitly utilize the relevance judgments existing with standard test collections in order to train the IR algorithms and possibly enhance retrieval accuracy for the new ( unseen ) queries .

Cohen , Shapire and Singer ( 1999 ) noted the differences between ordering and classification and presented a two-stage model to learn ordering .

The first stage learns a classifier for preference relations between objects using any suitable learning mechanism ( e.g. support vector machines ; Vapnik , 1998 ) .

The second stage converts preference relations into a rank order .

Although the conversion may be NP complete in a general case , they presented efficient approximations .

We limited our first study reported here to linear classifiers , in which conversion can be performed by simple ordering according to the score of each document .

However , approaching the problem as " learning how to order things " allowed us to design our sampling and training mechanisms in a novel and , we believe , more powerful way .

Our classifier learns how to compare every pair of documents with respect to a given query , based on the relevance indicating features that the documents may have .

As it is commonly done in information retrieval , the features are derived from the word overlap between the query and documents .

According to Nallapati ( 2004 ) , the earliest formulation of the classic IR problem as a classification ( discrimination ) problem was suggested by Robertson and Sparck Jones ( 1976 ) , however performed well only when the relevance judgments were available for the same query but not generalizing well to new queries .

Fuhr and Buckley ( 1991 ) used polynomial regression to estimate the coefficients in a linear scoring function combining such well-known features as a weighted term frequency , document length and query length .

They tested their " description-oriented " approach on the standard small-scale collections ( Cranfield , NPL , INSPEC , CISI , CACM ) to achieve the relative change in the average precision ranging from -17 % to + 33 % depending on the collection tested and the implementation parameters .

Gey ( 1994 ) applied logistic regression in a similar setting with the following results : Cranfield +12 % , CACM

-4.4 % , however he did not test them on new ( unseen by the algorithm ) queries , hypothesizing that splitting documents into training and testing collections would not be possible since " a large number of queries is necessary in order to train for a decent logistic regression approach to document retrieval . "

Instead , he applied a regression trained on Cranfield to CISI collection but with a negative effect .

Recently , the approaches based on learning have reported several important breakthroughs .

Fan et al . ( 2004 ) applied genetic programming in order to learn how to combine various terms into the optimal ranking function that outperformed the popular Okapi formula on robust retrieval test collection .

Nallapati ( 2004 ) made a strong argument in favor of discriminative models and trained an SVM-based classifier to combine 6 different components ( terms ) from the popular ranking functions ( such as tf.idf and language models ) to achieve better than the language model performance in 2 out of 16 test cases ( figure 3 in Nallapati , 2004 ) , not statistically distinguishable in 8 cases and only 80 % of the best performance in 6 cases .

There have been studies using past relevance judgements to optimize retrieval .

For example , Joachims ( 2002 ) applied Support Vector Machines to learn linear ranking function from user click-throughs while interfacing with a search engine .

In this study , we have developed a representation scheme , which is based on the discretization of the global ( corpus statistics ) and local ( document statistics ) weighting of term overlaps between queries and documents .

We have empirically shown that this representation is flexible enough to learn the properties of the popular ranking functions : tf.idf , BM25 and the language models .

The major difference of our work from Fan et al . ( 2004 ) or Nallapati ( 2004 ) or works on fusion ( e.g. Vogt & Cottrell , 1999 ) is that we did not try to combine several known ranking functions ( or their separate terms ) into one , but rather we learn the weighting functions directly through discretization .

Discretization allows representing a continuous function by a set of values at certain points .

These values are learned by a machine learning technique to optimize certain criteria , e.g. average precision .

Another important motivation behind using discretization was to design a representation with high dimensionality of features in order to combine our representation scheme with Support Vector Machines ( SVM ) ( Vapnik , 1998 ) , which are known to work well with a large number of features .

SVM contains a large class of neural nets , radial margin separation ( RBF ) nets , and polynomial classifiers as special cases .

They have been delivering superior performance in classification tasks in general domains , e.g. in face recognition ( Hearst , 1998 ) , and in text categorization ( Joachims , 2001 ) .

Another important distinction of this work from the prior research is that we train our classifier not to predict the absolute relevance of a document d with respect to a query q , but rather to predict which of the two documents d1 , d2 is more relevant to the query q.

The motivation for this distinction was that all the popular evaluation metrics in information retrieval ( e.g. average precision ) are based on document ranking rather than classification accuracy .

This affected our specially designed sampling procedure which we empirically discovered to be crucial for successful learning .

We have also empirically established that our combination of the representation scheme , learning mechanism and sampling allows learning from the past relevance judgments in order to successfully generalize to the new ( unseen ) queries .

When the representation was created without any knowledge of the top ranking functions and their parameters , our approach reached the known top performance solely through the learning process .

When our representation was taking advantage of functions that are known to perform well and their parameters , the resulting combination was able to slightly exceed the top performance on large test collections .

The next section formalizes our Discretization Based Learning ( DBL ) approach to Information Retrieval , followed by empirical results and conclusions .

We explored learning how to rank documents with respect to a given query using linear Support Vector Machines and discretization-based representation .

Our approach represents a family of discriminative approaches , currently studied much less than heuristic ( tf.idf , bm25 ) or generative approaches ( language models ) .

Our experiments indicate that learning from relevant judgments available with the standard test collections and generalizing to new queries is not only feasible but can be a source of improvement .

When tested with a popular standard collection , our approach achieved the performance of the best well-known techniques ( BM25 and language models ) , which have been developed as a result of extensive past experiments and elaborate theoretical modeling .

When combined with the best performing ranking functions , our approach added a small ( 2 - 3 % ) , but statistically significant , improvement .

Although practical significance of this study may be limited at the moment since it does not demonstrate a dramatic increase in retrieval performance in large test collections , we believe our findings have important theoretical contributions since they indicate that the power of discriminative approach is comparable to the best known analytical or heuristic apporaches .

This work also lays the foundation for extending the discriminative approach to " richer " representations , such as those using word n-grams , grammatical relations between words , and the structure of documents .

Our results also indicate that gw.lw family , which includes practically all the popular " bag of words " ranking formulas such as tf.idf , BM25 or language models , has almost reached its upper limit and other classes of representations and ranking formulas need to be explored in order to accomplish significant performance break-troughs .

Of course , using only few test cases ( topics sets and collections ) is a limitation of this current study , which we are going to address in our future research .

We view our approach as a complement , rather than competitive , to the analytical approaches such as language models .

Our approach can be also used as an explorative tool in order to identify important relevance-indicating features , which can be later modeled analytically .

We believe that our work and the ones referred in this paper may bring many of the achievements made in a more general area of classification and machine learning closer to the task of rank ordered information retrieval , thus making retrieval engines more helpful in reducing the information overload and meeting people 's needs .


Existing topic-based multi-document approaches on automatic related work summarization rely heavily on human-engineered features to retrieve snippets from the references.

They propose a neural data-driven seq2seq summarizer with a joint context-driven attention mechanism which measures contextual relevances within full texts and a heterogeneous bibliography graph.

The proposed model outperforms a typical seq2seq and five classical baseline models on experiments with 8080 papers with native related work sections.

Abstract: Conventional solutions to automatic related work summarization rely heavily on humanengineered features .

Abstract: In this paper , we develop a neural data-driven summarizer by leveraging the seq2seq paradigm , in which a joint context-driven attention mechanism is proposed to measure the contextual relevance within full texts and a heterogeneous bibliography graph simultaneously .

Abstract: Our motivation is to maintain the topic coherency between a related work section and its target document , where both the textual and graphic contexts play a big role in characterizing the relationship among scientific publications accurately .

Abstract: Experimental results on a large dataset show that our approach achieves a considerable improvement over a typical seq2seq summarizer and five classical summarization baselines .

In scientific fields , scholars need to contextualize their contribution to help readers acquire an understanding of their research papers .

For this purpose , the related work section of an article serves as a pivot to connect prior domain knowledge , in which the innovation and superiority of current work are displayed by a comparison with previous studies .

While citation prediction can assist in drafting a reference collection ( Nallapati et al . , 2008 ) , consuming all these papers is still a laborious job , where authors must read every source document carefully and locate the most relevant content cautiously .

As a solution in saving authors ' efforts , automatic related work summarization is essentially a topic-biased multi-document problem ( Cong and Kan , 2010 ) , which relies heavily on humanengineered features to retrieve snippets from the references .

Most recently , neural networks enable a data-driven architecture sequence-to-sequence ( seq2seq ) for natural language generation ( Bahdanau et al . , 2014 ( Bahdanau et al . , , 2016 ) ) , where an encoder reads a sequence of words / sentences into a context vector , from which a decoder yields a sequence of specific outputs .

Nonetheless , compared to scenarios like machine translation with an end-to-end nature , aligning a related work section to its source documents is far more challenging .

To address the summarization alignment , former studies try to apply an attention mechanism to measure the saliency / novelty of each candidate word / sentence ( Tan et al . , 2017 ) , with the aim of locating the most representative content to retain primary coverage .

However , toward summarizing a related work section , authors should be more creative when organizing text streams from the reference collection , where the selected content ought to highlight the topic bias of current work , rather than retell each reference in a compressed but balanced fashion .

This motivates us to introduce the contextual relevance and characterize the relationship among scientific publications accurately .

Generally speaking , for a pair of documents , a larger lexical overlap often implies a higher similarity in their research backgrounds .

Yet such a hypothesis is not always true when sampling content from multiple relevant topics .

Take " DSSM " 1 as an example , from viewpoint of the abstract similarity , those references investigating " Information Retrieval " , " Latent Semantic Model " or " Clickthrough Data Mining " could be of more importance in correlation and should be greatly sampled for the related work section .

But in reality , this article spends a bit larger chunk of texts ( about 58 % ) to elaborate " Deep Learning " during the literature review , which is quite difficult for machines to grasp the contextual relevance therein .

In addi-tion , other situations like emerging new concepts also suffer from the terminology variation or paraphrasing in varying degrees .

In this study , we utilize a heterogeneous bibliography graph to embody the relationship within a scalable scholarly database .

Over the recent past , there is a surge of interest in exploiting diverse relations to analyze bibliometrics , ranging from literature recommendation ( Yu et al . , 2015 ) to topic evolvement ( Jensen et al . , 2016 ) .

In a graphical sense , interconnected papers transfer the credit among each other directly / indirectly through various patterns , such as paper citation , author collaboration , keyword association and releasing on series of venues , which constitutes the graphic context for outlining concerned topics .

Unfortunately , a variety of edge types may pollute the information inquiry , where a slice of edges are not so important as the others on sampling content .

Meanwhile , most existing solutions in mining heterogeneous graphs depend on the human supervision , e.g. , hyperedge ( Bu et al . , 2010 ) and metapath ( Swami et al . , 2017 ) .

This is usually not easy to access due to the complexity of graph schemas .

First , we explore the edge-type usefulness distribution ( EUD ) on a heterogeneous bibliography graph , which enables the relationship discovery ( between any pair of papers ) for sampling the interested information .

Second , we develop a novel seq2seq summarizer for the automatic related work summarization , where a joint context-driven attention mechanism is proposed to measure the contextual relevance within both textual and graphic contexts .

Third , we conduct experiments on 8,080 papers with native related work sections , and experimental results show that our approach outperforms a typical seq2seq summarizer and five classical summarization baselines significantly .

In this paper , we highlight the contextual relevance for the automatic related work summarization , and analyze the graphic context to characterize the relationship among scientific publications accurately .

We develop a neural data-driven summarizer by leveraging the seq2seq paradigm , where a joint context-driven attention mechanism is proposed to measure the contextual relevance within full texts and a heterogeneous bibliography graph simultaneously .

Extensive experiments demonstrate the validity of the proposed attention mechanism , and the superiority of our approach over six representative summarization baselines .

In future work , an appealing direction is to organize the selected sentences in a logical fashion , e.g. , by leveraging a topic hierarchy tree to determine the arrangement of the related work section ( Cong and Kan , 2010 ) .

We also would like to take the citation sentences of each reference into consideration , which is another concise and universal data source for scientific summarization ( Chen and Hai , 2016 ;

At the end of this paper , we believe that extractive methods are by no means the final solutions for literature review generation due to plagiarism concerns , and we are going to put forward a fully abstractive version in further studies .


Existing studies on compositional generalization of neural models use synthetic data which limits the scope of analysis only to local expressions without global contexts considered.

They reformulate three theoretically grounded compositionality tests composed of systematicity, substitutivity for local and overgeneralisation for global on how models treat idioms.

They find that models trained on more data are more compositional but less compositional than expected and make non-compositional behaviours by mistake.

Abstract: Obtaining human-like performance in NLP is often argued to require compositional generalisation .

Abstract: Whether neural networks exhibit this ability is usually studied by training models on highly compositional synthetic data .

Abstract: However , compositionality in natural language is much more complex than the rigid , arithmeticlike version such data adheres to , and artificial compositionality tests thus do not allow us to determine how neural models deal with more realistic forms of compositionality .

Abstract: In this work , we re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation ( NMT ) .

Abstract: Our results highlight that : i ) unfavourably , models trained on more data are more compositional ; ii ) models are sometimes less compositional than expected , but sometimes more , exemplifying that different levels of compositionality are required , and models are not always able to modulate between them correctly ; iii ) some of the non-compositional behaviours are mistakes , whereas others reflect the natural variation in data .

Abstract: Apart from an empirical study , our work is a call to action : we should rethink the evaluation of compositionality in neural networks and develop benchmarks using real data to evaluate compositionality on natural language , where composing meaning is not as straightforward as doing the math .

Although the successes of deep neural networks in natural language processing ( NLP ) are astounding and undeniable , they are still regularly criticised for lacking the powerful generalisation capacities that characterise human intelligence .

A frequently mentioned concept in such critiques is compositionality : the ability to build up the meaning of a complex expression by combining the meanings of its parts ( e.g. Partee , 1984 ) .

Compositionality is assumed to play an essential role in how humans understand language , but whether neural networks also exhibit this property has since long been a topic of vivid debate ( e.g. Fodor and Pylyshyn , 1988 ; Smolensky , 1990 ; Marcus , 2003 ; Nefdt , 2020 ) .

Studies about the compositional abilities of neural networks consider almost exclusively models trained on synthetic datasets , in which compositionality can be ensured and isolated ( e.g. Lake and Baroni , 2018 ; Hupkes et al . , 2020 ) .

In such tests , the interpretation of expressions is computed completely locally : every subpart is evaluated independently -without taking into account any external context -and the meaning of the whole expression is then formed by combining the meanings of its parts in a bottom-up fashion .

This protocol matches the type of compositionality observed in arithmetic : the meaning of ( 3 + 5 ) is always 8 , independent of the context it occurs in .

However , as exemplified by the sub-par performance of symbolic models that allow only strict , local protocols , compositionality in natural domains is far more intricate than this rigid , arithmeticlike variant of compositionality .

Natural language seems very compositional , but at the same time , it is riddled with cases that are difficult to interpret with a strictly local interpretation of compositionality .

Sometimes , the meaning of an expression does not derive from its parts ( e.g. for idioms ) , but the parts themselves are used compositionally in other contexts .

In other cases , the meaning of an expression does depend on its parts in a compositional way , but arriving at this meaning requires a more global approach because the meanings of the parts need to be disambiguated by information from elsewhere .

For instance , consider the meaning of homonyms ( " these dates are perfect for our dish / wedding " ) , potentially idiomatic expressions ( " the child kicked the bucket off the pavement " ) , or scope ambiguities ( " every human likes a cat " ) .

This paradoxical tension between local and global forms of compositionality inspired many debates on the compositionality of natural language .

Likewise , it impacts the evaluation of compositionality in NLP models .

On the one hand , local compositionality seems necessary for robust and reliable generalisation .

Yet , at the same time , global compositionality is needed to appropriately address the full complexity of language , which makes evaluating compositionality of state-of-the-art models ' in the wild ' a complicated endeavour .

In this work , we face this challenge head-on .

We concentrate on the domain of neural machine translation ( NMT ) , which is paradigmatically close to the tasks typically considered for compositionality tests , where the target represents the ' meaning ' of the input .

Furthermore , MT is an important domain of NLP , for which compositional generalisation is important to produce more robust translations and train adequate models for low-resource languages ( see , e.g. Chaabouni et al . , 2021 ) .

As an added advantage , compositionality is traditionally well studied and motivated for MT ( Rosetta , 1994 ; Janssen and Partee , 1997 ; Janssen , 1998 ) .

We reformulate three theoretically grounded tests from Hupkes et al . ( 2020 ) : systematicity , substitutivity and overgeneralisation .

Since accuracycommonly used in artificial compositionality testsis not a suitable evaluation metric for MT , we base our evaluations on the extent to which models behave consistently , rather than correctly .

In our tests for systematicity and substitutivity , we consider whether processing is local ; in our overgeneralisation test , we consider how models treat idioms that are assumed to require global processing .

Our results indicate that models often do not behave compositionally under the local interpretation , but exhibit behaviour that is too local in other cases .

In other words , models have the ability to process phrases both locally and globally but do not always correctly modulate between them .

We further show that some inconsistencies reflect variation in natural language , whereas others are true compositional mistakes , exemplifying the need for both local and global compositionality as well as illustrating the need for tests that encompass them both .

With our study , we contribute to ongoing questions about the compositional abilities of neural networks , and we provide nuance to the nature of this question when natural language is concerned : how local should the compositionality of models for natural language actually be ?

Aside from an empirical study , our work is also a call to action : we should rethink the evaluation of compositionality in neural networks and develop benchmarks using real data to evaluate compositionality on natural language , where composing meaning is not as straightforward as doing the math .

In conclusion , with this work , we contribute to the question of how compositional models trained on natural data are , and we argue that MT is a suitable and relevant testing ground to ask this question .

Focusing on the balance between local and global forms of compositionality , we formulate three different compositionality tests and discuss the issues and considerations that come up when considering compositionality in the context of natural data .

Our tests indicate that models show both local and global processing , but not necessarily for the right samples .

Furthermore , they underscore the difficulty of separating helpful and harmful types of non-compositionality , stressing the need to rethink the evaluation of compositionality using natural language , where composing meaning is not as straightforward as doing the math .

The Npeople Vtransitive the N sl people .


Although existing works show that statistical approaches to dialogue management tasks outperform rule-based counterparts, they are not used in real world applications.

They propose to use Markov Decision Process and reinforcement learning for domain selection in multi-domain spoken dialogue systems to achieve extensibility in domains.

The proposed system outperforms rule-based baseline systems with a hand-crafted policy on human-based evaluation of naturalness in domain switching tasks.

Abstract: This paper proposes a Markov Decision Process and reinforcement learning based approach for domain selection in a multidomain Spoken Dialogue System built on a distributed architecture .

Abstract: In the proposed framework , the domain selection problem is treated as sequential planning instead of classification , such that confirmation and clarification interaction mechanisms are supported .

Abstract: In addition , it is shown that by using a model parameter tying trick , the extensibility of the system can be preserved , where dialogue components in new domains can be easily plugged in , without re-training the domain selection policy .

Abstract: The experimental results based on human subjects suggest that the proposed model marginally outperforms a non-trivial baseline .

Due to growing demand for natural humanmachine interaction , over the last decade Spoken Dialogue Systems ( SDS ) have been increasingly deployed in various commercial applications ranging from traditional call centre automation ( e.g. AT&T " Lets Go ! "

bus information system ( Williams et al . , 2010 ) ) to mobile personal assistants and knowledge navigators ( e.g. Apple 's Siri R , Google Now TM , Microsoft Cortana , etc . ) or voice interaction for smart household appliance control ( e.g. Samsung Evolution Kit for Smart TVs ) .

Furthermore , latest progress in openvocabulary Automatic Speech Recognition ( ASR ) is pushing SDS from traditional single-domain information systems towards more complex multidomain speech applications , of which typical examples are those voice assistant mobile applications .

Recent advances in SDS have shown that statistical approaches to dialogue management can result in marginal improvement in both the naturalness and the task success rate for domainspecific dialogues ( Lemon and Pietquin , 2012 ; Young et al . , 2013 ) .

State-of-the-art statistical SDS treat the dialogue problem as a sequential decision making process , and employ established planning models , such as Markov Decision Processes ( MDPs ) ( Singh et al . , 2002 ) or Partially Observable Markov Decision Processes ( POMDPs ) ( Thomson and Young , 2010 ; Young et al . , 2010 ; Williams and Young , 2007 ) , in conjunction with reinforcement learning techniques ( Jurčíček et al . , 2011 ; Jurčíček et al . , 2012 ; Gašić et al . , 2013a ) to seek optimal dialogue policies that maximise long-term expected ( discounted ) rewards and are robust to ASR errors .

However , to the best of our knowledge , most of the existing multi-domain SDS in public use are rule-based ( e.g. ( Gruber et al . , 2012 ; Mirkovic and Cavedon , 2006 ) ) .

The application of statistical models in multi-domain dialogue systems is still preliminary .

Komatani et al . ( 2006 ) and Nakano et al . ( 2011 ) utilised a distributed architecture ( Lin et al . , 1999 ) to integrate expert dialogue systems in different domains into a unified framework , where a central controller trained as a data-driven classifier selects a domain expert at each turn to address user 's query .

Alternatively , Hakkani-Tür et al . ( 2012 ) adopted the well-known Information State mechanism ( Traum and Larsson , 2003 ) to construct a multi-domain SDS and proposed a discriminative classification model for more accurate state updates .

More recently , Gašić et al . ( 2013b ) proposed that by a simple expansion of the kernel function in Gaussian Process ( GP ) reinforcement learning ( Engel et al . , 2005 ; Gašić et al . , 2013a ) , one can adapt pre-trained dialogue policies to handle unseen slots for SDS in extended domains .

In this paper , we use a voice assistant applica-

Figure 1 : The distributed architecture of the voice assistant system ( a simplified illustration ) .

tion ( similar to Apple 's Siri but in Chinese language ) as an example to demonstrate a novel MDP-based approach for central interaction management in a complex multi-domain dialogue system .

The voice assistant employs a distributed architecture similar to ( Lin et al . , 1999 ; Komatani et al . , 2006 ; Nakano et al . , 2011 ) , and handles mixed interactions of multi-turn dialogues across different domains and single-turn queries powered by a collection of information access services ( such as web search , Question Answering ( QA ) , etc . ) .

In our system , the dialogues in each domain are managed by an individual domain expert SDS , and the single-turn services are used to handle those so-called out-of-domain requests .

We use featurised representations to summarise the current dialogue states in each domain ( see Section 3 for more details ) , and let the central controller ( the MDP model ) choose one of the following system actions at each turn : ( 1 ) addressing user 's query based on a domain expert , ( 2 ) treating it as an out-of-domain request , ( 3 ) asking user to confirm whether he / she wants to continue a domain expert 's dialogue or to switch to out-of-domain services , and ( 4 ) clarifying user 's intention between two domains .

The Gaussian Process Temporal Difference ( GPTD ) algorithm ( Engel et al . , 2005 ; Gašić et al . , 2013a ) is adopted here for policy optimisation based on human subjects , where a parameter tying trick is applied to preserve the extensibility of the system , such that new domain experts ( dialogue systems ) can be flexibly plugged in without the need of re-training the central controller .

Comparing to the previous classification-based methods ( Komatani et al . , 2006 ; Nakano et al . , 2011 ) , the proposed approach not only has the advantage of action selection in consideration of long-term rewards , it can also yield more robust policies that allow clarifications and confirmations to mitigate ASR and Spoken Language Understanding ( SLU ) errors .

Our human evaluation results show that the proposed system with a trained MDP policy achieves significantly better naturalness in domain switching tasks than a non-trivial baseline with a hand-crafted policy .

The remainder of this paper is organised as follows .

Section 2 defines the terminology used throughout the paper .

Section 3 briefly overviews the distributed architecture of our system .

The MDP model and the policy optimisation algorithm are introduced in Section 4 and Section 5 , respectively .

After this , experimental settings and evaluation results are described in Section 6 .

Finally , we discuss some possible improvements in Section 7 and conclude ourselves in Section 8 .

In this paper , we introduce an MDP framework for learning domain selection policies in a complex multi-domain SDS .

Standard problem formulation is modified with tied model parameters , so that the entire system is extensible and new domain experts can be easily integrated without re-training the policy .

This expectation is confirmed by empirical experiments with human subjects , where the proposed system marginally beats a non-trivial baseline and demonstrates proper extensibility .

Several possible improvements are discussed , which will be the central arc of our future research .

Thomas Robert Gruber , Adam John Cheyer , Dag


Although there are varieties in the Chinese language, current lexical resources based on one variant leave questions of compatibility to others when to be extended.

They sample 30 finance-related words from several Chinese languages and evaluate if a classifier can match them to correct categories in well studied variant.

The evaluation with human judgements shows that classification accuracy reaches up to 85% and also finds that the performance depends on word sources.

Abstract: In this paper , we address a unique problem in Chinese language processing and report on our study on extending a Chinese thesaurus with region-specific words , mostly from the financial domain , from various Chinese speech communities .

Abstract: With the larger goal of automatically constructing a Pan-Chinese lexical resource , this work aims at taking an existing semantic classificatory structure as leverage and incorporating new words into it .

Abstract: In particular , it is important to see if the classification could accommodate new words from heterogeneous data sources , and whether simple similarity measures and clustering methods could cope with such variation .

Abstract: We use the cosine function for similarity and test it on automatically classifying 120 target words from four regions , using different datasets for the extraction of feature vectors .

Abstract: The automatic classification results were evaluated against human judgement , and the performance was encouraging , with accuracy reaching over 85 % in some cases .

Abstract: Thus while human judgement is not straightforward and it is difficult to create a Pan-Chinese lexicon manually , it is observed that combining simple clustering methods with the appropriate data sources appears to be a promising approach toward its automatic construction .

Large-scale semantic lexicons are important resources for many natural language processing ( NLP ) tasks .

For a significant world language such as Chinese , it is especially critical to capture the substantial regional variation as an important part of the lexical knowledge , which will be useful for many NLP applications , including natural language understanding , information retrieval , and machine translation .

Existing Chinese lexical resources , however , are often based on language use in one particular region and thus lack the desired comprehensiveness .

Toward this end , Tsou and Kwong ( 2006 ) proposed a comprehensive Pan-Chinese lexical resource , based on a large and unique synchronous Chinese corpus as an authentic source for lexical acquisition and analysis across various Chinese speech communities .

To allow maximum versatility and portability , it is expected to document the core and universal substances of the language on the one hand , and also the more subtle variations found in different communities on the other .

Different Chinese speech communities might share lexical items in the same form but with different meanings .

For instance , the word 居屋 refers to general housing in Mainland China but specifically to housing under the Home Ownership Scheme in Hong Kong ; and while the word 住房 is similar to 居屋 to mean general housing in Mainland China , it is rarely seen in the Hong Kong context .

Hence , the current study aims at taking an existing Chinese thesaurus , namely the Tongyici Cilin 同義詞詞林 , as leverage and extending it with lexical items specific to individual Chinese speech communities .

In particular , the feasibility depends on the following issues : ( 1 ) Can lexical items from various Chinese speech communities , that is , from such heterogeneous sources , be classified as effectively with methods shown to work for clustering closely related words from presumably the same , or homogenous , source ?

( 2 ) Could existing semantic classificatory structures accommodate concepts and expressions specific to individual Chinese speech communities ?

Measuring similarity will make sense only if the feature vectors of the two words under comparison are directly comparable .

There is usually no problem if both words and their contextual features are from the same data source .

Since Tongyici Cilin ( or simply Cilin hereafter ) is based on the vocabulary used in Mainland China , it is not clear how often these words will be found in data from other places , and even if they are found , how well the feature vectors extracted could reflect the expected usage or sense .

Our hypothesis is that it will be more effective to classify new words from Mainland China with respect to Cilin categories , than to do the same on new words from regions outside Mainland China .

Furthermore , if this hypothesis holds , one would need to consider separate mechanisms to cluster heterogeneous regionspecific words in the Pan-Chinese context .

Thus in the current study we sampled 30 target words specific to each of Beijing , Hong Kong , Singapore , and Taipei , from the financial domain ; and used the cosine similarity function to classify them into one or more of the semantic categories in Cilin .

The automatic classification results were compared with a simple baseline method , against human judgement as the gold standard .

In general , an accuracy of up to 85 % could be reached with the top 15 candidates considered .

It turns out that our hypothesis is supported by the Taipei test data , whereas the data heterogeneity effect is less obvious in Hong Kong and Singapore test data , though the effect on individual test items varies .

In Section 2 , we will briefly review related work and highlight the innovations of the current study .

In Sections 3 and 4 , we will describe the materials used and the experimental setup respectively .

Results will be presented and discussed with future directions in Section 5 , followed by a conclusion in Section 6 .

In this paper , we have reported our study on a unique problem in Chinese language processing , namely extending a Chinese thesaurus with new words from various Chinese speech communities , including Beijing , Hong Kong , Singapore and Taipei .

The critical issues include whether the existing classificatory structure could accommodate concepts and expressions specific to various Chinese speech communities , and whether the difference in textual sources might pose difficulty in using conventional similarity measures for the automatic classification .

Our experiments , using the cosine function to measure similarity and testing with various sources for extracting contextual vectors , suggest that the classification performance might depend on the compatibility between the words in the thesaurus and the sources from which the target words are drawn .

Evaluated against human judgement , an accuracy of over 85 % was reached in some cases , which were much higher than the baseline and were very encouraging in general .

While human judgement is not straightforward and it is difficult to create a Pan-Chinese lexicon manually , combining simple classification methods with the appropriate data sources seems to be a promising approach toward its automatic construction .


Product review data is not structured hindering users from understanding its overview, and existing methods that build hierarchical structures suffer from inconsistency or low accuracy.

They propose to organize product aspects and formulize them into a hierarchical structure by exploiting existing domain knowledge-bases such as WordNet to navigate users.

The proposed method that converts reviews into hierarchical structures exhibits effectiveness in experiments with 11 popular products from four domains.

Abstract: This paper presents a domain-assisted approach to organize various aspects of a product into a hierarchy by integrating domain knowledge ( e.g. , the product specifications ) , as well as consumer reviews .

Abstract: Based on the derived hierarchy , we generate a hierarchical organization of consumer reviews on various product aspects and aggregate consumer opinions on these aspects .

Abstract: With such organization , user can easily grasp the overview of consumer reviews .

Abstract: Furthermore , we apply the hierarchy to the task of implicit aspect identification which aims to infer implicit aspects of the reviews that do not explicitly express those aspects but actually comment on them .

Abstract: The experimental results on 11 popular products in four domains demonstrate the effectiveness of our approach .

With the rapidly expanding e-commerce , most retail Web sites encourage consumers to write reviews to express their opinions on various aspects of products .

Huge collections of consumer reviews are now available on the Web .

These reviews have become an important resource for both consumers and firms .

Consumers commonly seek quality information from online consumer reviews prior to purchasing a product , while many firms use online reviews as an important resource in their product development , marketing , and consumer relationship management .

However , the reviews are disorganized , leading to the difficulty in information navigation and knowledge acquisition .

It is impractical for user to grasp the overview of consumer reviews and opinions on various aspects of a product from such enormous reviews .

Among hundreds of product aspects , it is also inefficient for user to browse consumer reviews and opinions on a specific aspect .

Thus , there is a compelling need to organize consumer reviews , so as to transform the reviews into a useful knowledge structure .

Since the hierarchy can improve information representation and accessibility ( Cimiano , 2006 ) , we propose to organize the aspects of a product into a hierarchy and generate a hierarchical organization of consumer reviews accordingly .

Towards automatically deriving an aspect hierarchy from the reviews , we could refer to traditional hierarchy generation methods in ontology learning , which first identify concepts from the text , then determine the parent-child relations between these concepts using either pattern-based or clusteringbased methods ( Murthy et al . , 2010 ) .

However , pattern-based methods usually suffer from inconsistency of parent-child relationships among the concepts , while clustering-based methods often result in low accuracy .

Thus , by directly utilizing these methods to generate an aspect hierarchy from consumer reviews , the resulting hierarchy is usually inaccurate , leading to unsatisfactory review organization .

On the other hand , domain knowledge of products is now available on the Web .

For example , there are more than 248,474 product specifications in the product selling Web site CNet.com ( Beckham , 2005 ) .

These product specifications cover some product aspects and provide coarse-grained parentchild relations among these aspects .

Such domain knowledge is useful to help organize the product as-140 pects into a hierarchy .

However , the initial hierarchy obtained from domain knowledge usually can not fit the review data well .

For example , the initial hierarchy is usually too coarse and may not cover the specific aspects commented in the reviews , while some aspects in the hierarchy may not be of interests to users in the reviews .

Motivated by the above observations , we propose in this paper to organize the product aspects into a hierarchy by simultaneously exploiting the domain knowledge ( e.g. , the product specification ) and consumer reviews .

With derived aspect hierarchy , we generate a hierarchical organization of consumer reviews on various aspects and aggregate consumer opinions on these aspects .

Figure 1 illustrates a sample of hierarchical review organization for the product " iPhone 3 G " .

With such organization , users can easily grasp the overview of product aspects as well as conveniently navigate the consumer reviews and opinions on any aspect .

For example , users can find that 623 reviews , out of 9,245 reviews , are about the aspect " price " , with 241 positive and 382 negative reviews .

Given a collection of consumer reviews on a specific product , we first automatically acquire an initial aspect hierarchy from domain knowledge and identify the aspects from the reviews .

Based on the initial hierarchy , we develop a multi-criteria optimization approach to construct an aspect hierarchy to contain all the identified aspects .

Our approach incrementally inserts the aspects into the initial hierarchy based on inter-aspect semantic distance , a metric used to measure the semantic relation among aspects .

In order to derive reliable semantic distance , we propose to leverage external hierarchies , sampled from WordNet and Open Directory Project , to assist semantic distance learning .

With resultant aspect hierarchy , the consumer reviews are then organized to their corresponding aspect nodes in the hierarchy .

We then perform sentiment classification to determine consumer opinions on these aspects .

Furthermore , we apply the hierarchy to the task of implicit aspect identification .

This task aims to infer implicit aspects of the reviews that do not explicitly express those aspects but actually comment on them .

For example , the implicit aspect of the review " It is so expensive " is " price . "

Most existing aspect identification approaches rely on the appearance of aspect terms , and thus are not able to handle implicit aspect problem .

Based on our aspect hierarchy , we can infer the implicit aspects by clustering the reviews into their corresponding aspect nodes in the hierarchy .

We conduct experiments on 11 popular products in four domains .

More details of the corpus are discussed in Section 4 .

The experimental results demonstrate the effectiveness of our approach .

The main contributions of this work can be summarized as follows :

1 ) We propose to hierarchically organize consumer reviews according to an aspect hierarchy , so as to transfer the reviews into a useful knowledge structure .

2 ) We develop a domain-assisted approach to generate an aspect hierarchy by integrating domain knowledge and consumer reviews .

In order to derive reliable semantic distance between aspects , we propose to leverage external hierarchies to assist semantic distance learning .

3 ) We apply the aspect hierarchy to the task of implicit aspect identification , and achieve satisfactory performance .

The rest of this paper is organized as follows .

Our approach is elaborated in Section 2 and applied to implicit aspect identification in Section 3 .

Section 4 presents the evaluations , while Section 5 reviews 141 related work .

Finally , Section 6 concludes this paper with future works .

In this paper , we have developed a domain-assisted approach to generate product aspect hierarchy by integrating domain knowledge and consumer reviews .

Based on the derived hierarchy , we can generate a hierarchical organization of consumer reviews as well as consumer opinions on the aspects .

With such organization , user can easily grasp the overview of consumer reviews , as well as seek consumer reviews and opinions on any specific aspect by navigating through the hierarchy .

We have further applied the hierarchy to the task of implicit aspect identification .

We have conducted evaluations on 11 different products in four domains .

The experimental results have demonstrated the effectiveness of our approach .

In the future , we will explore other linguistic features to learn the semantic distance between aspects , as well as apply our approach to other applications .


Existing methods for learning word representations are restricted to a single language.

They extend the distributional hypothesis to multilingual data and joint-space embeddings and use parallel data to obtain multilingual word representations.

Multilingual word vectors learned using the proposed model outperform existing methods on cross-lingual document classification tasks without word-level alignment or syntactic information in diverse languages.

Abstract: We present a novel technique for learning semantic representations , which extends the distributional hypothesis to multilingual data and joint-space embeddings .

Abstract: Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences , while maintaining sufficient distance between those of dissimilar sentences .

Abstract: The models do not rely on word alignments or any syntactic information and are successfully applied to a number of diverse languages .

Abstract: We extend our approach to learn semantic representations at the document level , too .

Abstract: We evaluate these models on two cross-lingual document classification tasks , outperforming the prior state of the art .

Abstract: Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic relationships across languages without parallel data .

Distributed representations of words provide the basis for many state-of-the-art approaches to various problems in natural language processing today .

Such word embeddings are naturally richer representations than those of symbolic or discrete models , and have been shown to be able to capture both syntactic and semantic information .

Successful applications of such models include language modelling ( Bengio et al . , 2003 ) , paraphrase detection ( Erk and Padó , 2008 ) , and dialogue analysis ( Kalchbrenner and Blunsom , 2013 ) .

Within a monolingual context , the distributional hypothesis ( Firth , 1957 ) forms the basis of most approaches for learning word representations .

In this work , we extend this hypothesis to multilingual data and joint-space embeddings .

We present a novel unsupervised technique for learning semantic representations that leverages parallel corpora and employs semantic transfer through compositional representations .

Unlike most methods for learning word representations , which are restricted to a single language , our approach learns to represent meaning across languages in a shared multilingual semantic space .

First , we show that for cross-lingual document classification on the Reuters RCV1 / RCV2 corpora ( Lewis et al . , 2004 ) , we outperform the prior state of the art ( Klementiev et al . , 2012 ) .

Second , we also present classification results on a massively multilingual corpus which we derive from the TED corpus ( Cettolo et al . , 2012 ) .

The results on this task , in comparison with a number of strong baselines , further demonstrate the relevance of our approach and the success of our method in learning multilingual semantic representations over a wide range of languages .

To summarize , we have presented a novel method for learning multilingual word embeddings using parallel data in conjunction with a multilingual objective function for compositional vector models .

This approach extends the distributional hypothesis to multilingual joint-space representations .

Coupled with very simple composition functions , vectors learned with this method outperform the state of the art on the task of cross-lingual document classification .

Further experiments and analysis support our hypothesis that bilingual signals are a useful tool for learning distributed representations by enabling models to abstract away from mono-lingual surface realisations into a deeper semantic space .


Applying syntactic pre-ordering to statistical machine translation is effective especially for language pairs with different word orders, however; current methods require a supervised syntactic parser.

they propose an unsupervised method that can build parsers for pre-ordering using a word-aligned parallel corpus without syntactic annotations.

The pre-ordering parsers obtained by the proposed method can perform better reordering than state-of-the-art methods and are also effective with machine translation systems.

Abstract: When translating among languages that differ substantially in word order , machine translation ( MT ) systems benefit from syntactic preordering-an approach that uses features from a syntactic parse to permute source words into a target-language-like order .

Abstract: This paper presents a method for inducing parse trees automatically from a parallel corpus , instead of using a supervised parser trained on a treebank .

Abstract: These induced parses are used to preorder source sentences .

Abstract: We demonstrate that our induced parser is effective : it not only improves a state-of-the-art phrase-based system with integrated reordering , but also approaches the performance of a recent preordering method based on a supervised parser .

Abstract: These results show that the syntactic structure which is relevant to MT pre-ordering can be learned automatically from parallel text , thus establishing a new application for unsupervised grammar induction .

Recent work in statistical machine translation ( MT ) has demonstrated the effectiveness of syntactic preordering : an approach that permutes source sentences into a target-like order as a pre-processing step , using features of a source-side syntactic parse ( Collins et al . , 2005 ; Xu et al . , 2009 ) .

Syntactic pre-ordering is particularly effective at applying structural transformations , such as the ordering change from a subject-verb-object ( SVO ) language like English to a subject-object-verb ( SOV ) language like Japanese .

However , state-of-the-art pre-ordering methods require a supervised syntactic parser to provide structural information about each sentence .

We propose a method that learns both a parsing model and a reordering model directly from a word-aligned parallel corpus .

Our approach , which we call Structure Induction for Reordering ( STIR ) , requires no syntactic annotations to train , but approaches the performance of a recent syntactic pre-ordering method in a large-scale English-Japanese MT system .

STIR predicts a pre-ordering via two pipelined models : ( 1 ) parsing and ( 2 ) tree reordering .

The first model induces a binary parse , which defines the space of possible reorderings .

In particular , only trees that properly separate verbs from their object noun phrases will license an SVO to SOV transformation .

The second model locally permutes this tree .

Our approach resembles work with binary synchronous grammars ( Wu , 1997 ) , but is distinct in its emphasis on monolingual parsing as a first phase , and in selecting reorderings without the aid of a target-side language model .

The parsing model is trained to maximize the conditional likelihood of trees that license the reorderings implied by observed word alignments in a parallel corpus .

This objective differs from those of previous grammar induction models , which typically focus on succinctly explaining the observed source language corpus via latent hierarchical structure ( Pereira and Schabes , 1992 ; Klein and Manning , 2002 ) .

Our convex objective allows us to train a feature-rich log-linear parsing model , even without supervised treebank data .

Focusing on pre-ordering for MT leads to a new perspective on the canonical NLP task of grammar induction-one which marries the wide-spread scientific interest in unsupervised parsing models with a clear application and extrinsic evaluation methodology .

To support this perspective , we highlight several avenues of future research throughout the paper .

We evaluate STIR in a large-scale English-Japanese machine translation system .

We measure how closely our predicted reorderings match those implied by hand-annotated word alignments .

STIR approaches the performance of the state-of-the-art pre-ordering method described in Genzel ( 2010 ) , which learns reordering rules for supervised treebank parses .

STIR gives a translation improvement of 3.84 BLEU over a standard phrase-based system with an integrated reordering model .

We have demonstrated that induced parses suffice for pre-ordering .

We hope that future work in grammar induction will also consider pre-ordering as an extrinsic evaluation .


Existing neural network-based models with embedding concatenation cannot process complex discrete features, and larger models have problems such as overfitting, and underperforming simple linear systems.

They propose a gated recursive neural network with two novel gating mechanisms trained with layer-wise training method and dropout to mitigate gradient diffusion and overfitting.

The proposed gated recursive neural network outperforms traditional linear models with feature engineering and also neural network-based models on the task of Chinese word segmentation.

Abstract: Recently , neural network models for natural language processing tasks have been increasingly focused on for their ability of alleviating the burden of manual feature engineering .

Abstract: However , the previous neural models can not extract the complicated feature compositions as the traditional methods with discrete features .

Abstract: In this paper , we propose a gated recursive neural network ( GRNN ) for Chinese word segmentation , which contains reset and update gates to incorporate the complicated combinations of the context characters .

Abstract: Since GRNN is relative deep , we also use a supervised layer-wise training method to avoid the problem of gradient diffusion .

Abstract: Experiments on the benchmark datasets show that our model outperforms the previous neural network models as well as the state-of-the-art methods .

Unlike English and other western languages , Chinese do not delimit words by white-space .

Therefore , word segmentation is a preliminary and important pre-process for Chinese language processing .

Most previous systems address this problem by treating this task as a sequence labeling problem and have achieved great success .

Due to the nature of supervised learning , the performance of these models is greatly affected by the design of features .

These features are explicitly represented by the different combinations of context characters , which are based on linguistic intuition and statistical information .

However , the number of features could be so large that the result models are too large to use in practice and prone to overfit on training corpus .

Figure 1 : Illustration of our model for Chinese word segmentation .

The solid nodes indicate the active neurons , while the hollow ones indicate the suppressed neurons .

Specifically , the links denote the information flow , where the solid edges denote the acceptation of the combinations while the dashed edges means rejection of that .

As shown in the right figure , we receive a score vector for tagging target character " 地 " by incorporating all the combination information .

Recently , neural network models have been increasingly focused on for their ability to minimize the effort in feature engineering .

Collobert et al . ( 2011 ) developed a general neural network architecture for sequence labeling tasks .

Following this work , many methods ( Zheng et al . , 2013 ; Pei et al . , 2014 ; Qi et al . , 2014 ) applied the neural network to Chinese word segmentation and achieved a performance that approaches the state-of-the-art methods .

However , these neural models just concatenate the embeddings of the context characters , and feed them into neural network .

Since the concatenation operation is relatively simple , it is difficult to model the complicated features as the traditional discrete feature based models .

Although the complicated interactions of inputs can be modeled by the deep neural network , the previous neural model shows that the deep model can not outperform the one with a single non-linear model .

Therefore , the neural model only captures the interactions by the simple transition matrix and the single non-linear transformation .

These dense features extracted via these simple interactions are not nearly as good as the substantial discrete features in the traditional methods .

In this paper , we propose a gated recursive neural network ( GRNN ) to model the complicated combinations of characters , and apply it to Chinese word segmentation task .

Inspired by the success of gated recurrent neural network ( Chung et al . , 2014 ) , we introduce two kinds of gates to control the combinations in recursive structure .

We also use the layer-wise training method to avoid the problem of gradient diffusion , and the dropout strategy to avoid the overfitting problem .

Figure 1 gives an illustration of how our approach models the complicated combinations of the context characters .

Given a sentence " 雨 ( Rainy ) 天 ( Day ) 地面 ( Ground ) 积水 ( Accumulated water ) " , the target character is " 地 " .

This sentence is very complicated because each consecutive two characters can be combined as a word .

To predict the label of the target character " 地 " under the given context , GRNN detects the combinations recursively from the bottom layer to the top .

Then , we receive a score vector of tags by incorporating all the combination information in network .

The contributions of this paper can be summarized as follows :

• We propose a novel GRNN architecture to model the complicated combinations of the context characters .

GRNN can select and preserve the useful combinations via reset and update gates .

These combinations play a similar role in the feature engineering of the traditional methods with discrete features .

• We evaluate the performance of Chinese word segmentation on PKU , MSRA and CTB6 benchmark datasets which are commonly used for evaluation of Chinese word segmentation .

Experiment results show that our model outperforms other neural network models , and achieves state-of-the-art performance .

In this paper , we propose a gated recursive neural network ( GRNN ) to explicitly model the combinations of the characters for Chinese word segmentation task .

Each neuron in GRNN can be regarded as a different combination of the input characters .

Thus , the whole GRNN has an ability to simulate the design of the sophisticated features in traditional methods .

Experiments show that our proposed model outperforms the state-of-the-art methods on three popular benchmark datasets .

Despite Chinese word segmentation being a specific case , our model can be easily generalized and applied to other sequence labeling tasks .

In future work , we would like to investigate our proposed GRNN on other sequence labeling tasks .


Existing multi-modal representation learning approaches aim to obtain high-level summary embeddings that are useful for retrieval but difficult to compare encoder activations from different modalities.

They propose a semi-supervised learning framework with a discrete embedding space shared across modalities by vector quantization and an objective to match representations between modalities.

The proposed representations can complement high-level representations for retrieval and they use individual clusters to represent the same semantic concept across modalities demonstrating their interpretability.

Abstract: In contrast to recent advances focusing on highlevel representation learning across modalities , in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words .

Abstract: Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities .

Abstract: Beyond the shared embedding space , we propose a Cross-Modal Code Matching objective that forces the representations from different views ( modalities ) to have a similar distribution over the discrete embedding space such that cross-modal objects / actions localization can be performed without direct supervision .

Abstract: We show that the proposed discretized multi-modal finegrained representation ( e.g. , pixel / word / frame ) can complement high-level summary representations ( e.g. , video / sentence / waveform ) for improved performance on cross-modal retrieval tasks .

Abstract: We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities .

Toddlers acquire much of their knowledge through grounded learning -visual concepts can be acquired through language , and language acquisition emerges through visual interaction .

Inspired by this type of grounded learning , a rich body of representation learning research ( Harwath et al . , 2018 ; Miech et al . , 2020 ;

Alayrac et al . , 2020 ; Monfort et al . , 2021 ; Luo et al . , 2021 ) has been exploring the potential to learn from multi-modal data such as video-text , video-audio , and image-audio pairs .

These works typically focus on learning a joint embedding space between different modalities , in which high-level summary representations are extracted as embedding vectors .

These embedding vectors often represent entire video clips , spoken utterances , or sentences as single vectors , and can be useful on tasks such as cross-modal data retrieval , e.g. , finding the most similar visual scene according to a spoken language description .

The predominant approach to learning these embedding vectors is to use modality-independent encoders , and while this has been successful for downstream retrieval tasks , it makes it difficult to compare the activations of the encoders from different modalities .

Further , the space of continuous embedding vectors is unbounded , which makes interpreting the learned representations challenging .

To this end , we propose to jointly learn highlevel embedding vector representations with a finegrained discrete embedding space that is shared across different modalities .

The discrete embedding space enables model interpretability since there are a finite number of embedding vectors which are shared across modalities .

Besides the shared embedding space , we propose a Cross-Modal Code Matching ( CMCM ) objective that guides the embedding space to capture cross-modal correspondences of concepts , actions , and words .

This not only improves downstream performance on retrieval , but also allows us to better interpret what the model recognized through cross-modal grounded learning .

To verify the effectiveness of our proposed learning framework , we conducted experiments in several cross-modal domains , including video-text , video-audio , and image-audio .

We found consistent improvements over baseline models , verifying that the gain was not restricted to the particular choice of network architecture , input modalities , or dataset .

We also demonstrate the interpretability of the fine-grained discrete representations by showing the cross-modal relations between the embedding vectors and semantic concepts appearing in the input modalities .

Our approach also enables cross-modal concept localization without requiring any labels during training .

Figure 1 : An overview of the proposed framework .

The proposed shared discrete embedding space ( green region , described in Section 2.2 ) is based on a cross-modal representation learning paradigm ( blue / yellow regions , described in Section 2.1 ) .

The proposed Cross-Modal Code Matching L CMCM objective is detailed in Section 2.3 and Figure 2 .

In this paper , we proposed a framework for crossmodal representation learning with a discrete embedding space that is shared amongst different modalities and enables model interpretability .

We also propose a Cross-Modal Code Matching objective that encourages models to represent crossmodel semantic concepts in the embedding space .

Combining our discrete embedding space and objective with existing cross-modal representation learning models improves retrieval performance on video-text , video-audio , and image-audio datasets .

We also analyze the shared embedding space and find that semantically related video and audio inputs tend to use the same codewords .


Existing parsing methods for low-resource languages either use bootstrapping which requires parallel corpus that is not often available or a linguistically-motivated approach which lacks robustness.

They present a Gibbs sampler-based Bayesian approach with a grammar with latent annotations and its extension which learns rules from probabilities in an unsupervised PCFG.

The proposed technique performs on par or surpasses previous approaches in many languages but especially performs well in Kinyarwanda and Malagasy with small corpora.

Abstract: PCFGs with latent annotations have been shown to be a very effective model for phrase structure parsing .

Abstract: We present a Bayesian model and algorithms based on a Gibbs sampler for parsing with a grammar with latent annotations .

Abstract: For PCFG-LA , we present an additional Gibbs sampler algorithm to learn annotations from training data , which are parse trees with coarse ( unannotated ) symbols .

Abstract: We show that a Gibbs sampling technique is capable of parsing sentences in a wide variety of languages and producing results that are on-par with or surpass previous approaches .

Abstract: Our results for Kinyarwanda and Malagasy in particular demonstrate that low-resource language parsing can benefit substantially from a Bayesian approach .

Despite great progress over the past two decades on parsing , relatively little work has considered the problem of creating accurate parsers for low-resource languages .

Existing work in this area focuses primarily on approaches that use some form of cross-lingual bootstrapping to improve performance .

For instance , Hwa et al . ( 2005 ) use a parallel Chinese / English corpus and an English dependency grammar to induce an annotated Chinese corpus in order to train a Chinese dependency grammar .

Kuhn ( 2004b ) also considers the benefits of using multiple languages to induce a monolingual grammar , making use of a measure for data reliability in order to weight training data based on confidence of annotation .

Bootstrapping approaches such as these achieve markedly improved results , but they are dependent on the existence of a parallel bilingual corpus .

Very few such corpora are readily available , particularly for low-resource languages , and creating such corpora obviously presents a challenge for many practical applications .

Kuhn ( 2004a ) shows some of the difficulty in handling low-resource languages by examining various tasks using Q'anjob'al as an example .

Another approach is that of Bender et al . ( 2002 ) , who take a more linguistically-motivated approach by making use of linguistic universals to seed newly developed grammars .

This substantially reduces the effort by making it unnecessary to learn the basic parameters of a language , but it lacks the robustness of grammars learned from data .

Recent work on Probabilistic Context-Free Grammars with latent annotations ( PCFG-LA ) ( Matsuzaki et al . , 2005 ; Petrov et al . , 2006 ) have shown them to be effective models for syntactic parsing , especially when less training material is available ( Liang et al . , 2009 ; Shindo et al . , 2012 ) .

The coarse nonterminal symbols found in vanilla PCFGs are refined by latent variables ; these latent annotations can model subtypes of grammar symbols that result in better grammars and enable better estimates of grammar productions .

In this paper , we provide a Gibbs sampler for learning PCFG-LA models and show its effectiveness for parsing lowresource languages such as Malagasy and Kinyawanda .

Previous PCFG-LA work focuses on the problem of parameter estimation , including expectationmaximization ( EM ) ( Matsuzaki et al . , 2005 ; Petrov et al . , 2006 ) , spectral learning ( Cohen et al . , 2012 ; Cohen et al . , 2013 ) , and variational inference ( Liang et al . , 2009 ; Wang and Blunsom , 2013 ) .

Regardless of inference method , previous work has used the same method to parse new sentences : a Viterbi parse under a new sentence-specific PCFG obtained from an approximation of the original grammar ( Matsuzaki et al . , 2005 ) .

Here , we provide an alternative approach to parsing new sentences : an extension of the Gibbs sampling algorithm of Johnson et al . ( 2007 ) , which learns rule probabilities in an unsupervised PCFG .

We use a Gibbs sampler to collect sampled trees theoretically distributed from the true posterior distribution in order to parse .

Priors in a Bayesian model can control the sparsity of grammars ( which the insideoutside algorithm fails to do ) , while naturally incorporating smoothing into the model ( Johnson et al . , 2007 ; Liang et al . , 2009 ) .

We also build a Bayesian model for parsing with a treebank , and incorporate information from training data as a prior .

Moreover , we extend the Gibbs sampler to learn and parse PCFGs with latent annotations .

Learning the latent annotations is a compute-intensive process .

We show how a small amount of training data can be used to bootstrap : after running a large number of sampling iterations on a small set , the resulting parameters are used to seed a smaller number of iterations on the full training data .

This allows us to employ more latent annotations while maintaining reasonable training times and still making full use of the available training data .

To determine the cross-linguistic applicability of these methods , we evaluate on a wide variety of languages with varying amounts of available training data .

We use English and Chinese as examples of languages with high data availability , while Italian , Malagasy , and Kinyarwanda provide examples of languages with little available data .

We find that our technique comes near state of the art results on large datasets , such as those for Chinese and English , and it provides excellent results on limited datasets -both artificially limited in the case of English , and naturally limited in the case of Italian , Malagasy , and Kinyarwanda .

This , combined with its ability to run off-the-shelf on new languages without any supporting materials such as parallel corpora , make it a valuable technique for the parsing of low-resource languages .

Our experiments demonstrate that sampling vanilla PCFGs , as well as PCFGs with latent annotations , is feasible with the use of a Gibbs sampler technique and produces results that are in line with previous parsers on controlled test sets .

Our results also show that our methods are effective on a wide variety of languages-including two low-resource languageswith no language-specific model modifications needed .

Additionally , although not a uniform winner , the Gibbs-PCFG shows a propensity for performing well on naturally small corpora ( here , KIN / MLG ) .

The exact reason for this remains slightly unclear , but the fact that a similar advantage is not found for extremely small versions of large corpora indicates that our approach may be particularly well-suited for application in real low-resource environments as opposed to a sim-

Having established this procedure and its relative tolerance for low amounts of data , we would like to extend the model to make use of partial bracketing information instead of complete trees , perhaps in the form of Fragmentary Unlabeled Dependency Grammar annotations ( Schneider et al . , 2013 ) .

This would allow the sampling procedure to potentially operate using corpora with lighter annotations than full trees , making initial annotation effort not quite as heavy and potentially increasing the amount of available data for low-resource languages .

Additionally , using the expert partial annotations to help restrict the sample space could provide good gains in terms of training time .


They show that current statistical language models assume a relation between the backoff and the higher models although it is not always guaranteed.

They propose a generalization of linear interpolation for cases when the required relation by current methods is not satisfied.

The proposed method significantly improves the performance of a decision tree language model in concerned cases.

Abstract: In the face of sparsity , statistical models are often interpolated with lower order ( backoff ) models , particularly in Language Modeling .

Abstract: In this paper , we argue that there is a relation between the higher order and the backoff model that must be satisfied in order for the interpolation to be effective .

Abstract: We show that in n-gram models , the relation is trivially held , but in models that allow arbitrary clustering of context ( such as decision tree models ) , this relation is generally not satisfied .

Abstract: Based on this insight , we also propose a generalization of linear interpolation which significantly improves the performance of a decision tree language model .

A prominent use case for Language Models ( LMs ) in NLP applications such as Automatic Speech Recognition ( ASR ) and Machine Translation ( MT ) is selection of the most fluent word sequence among multiple hypotheses .

Statistical LMs formulate the problem as the computation of the model 's probability to generate the word sequence w 1 w 2 . . .

w m ≡ w m 1 , assuming that higher probability corresponds to more fluent hypotheses .

LMs are often represented in the following generative form :

In the following discussion , we will refer to the function p(w

Note the context space for this function , w i-1 1 is arbitrarily long , necessitating some independence assumption , which usually consists of reducing the relevant context to n -1 immediately preceding tokens :

These distributions are typically estimated from observed counts of n-grams w i i-n+1 in the training data .

The context space is still far too large ; therefore , the models are recursively smoothed using lower order distributions .

For instance , in a widely used n-gram LM , the probabilities are estimated as follows :

In addition to n-gram models , there are many other ways to estimate probability distributions p(w

i |w i-1 i-n+1 ) ; in this work , we are particularly interested in models involving decision trees ( DTs ) .

As in n-gram models , DT models also often utilize interpolation with lower order models ; however , there are issues concerning the interpolation which arise from the fact that decision trees permit arbitrary clustering of context , and these issues are the main subject of this paper .

The main contribution of this paper is the insight that in the standard recursive backoff there is an implied relation between the backoff and the higher order models , which is essential for adequate performance .

When this relation is not satisfied other interpolation methods should be employed ; hence , we propose a generalization of linear interpolation that significantly outperforms the standard form in such a scenario .


Incorporating syntactic information into RNN models to guide the composition of phrases is shown effective for classification tasks but not studied for other tasks.

They propose a supervised RNN model which uses syntactic information to explicitly weight important phrases for a semantic relation classification task.

Experiments show that using syntactic information such as phrase categories and heads, and also task-specific weighting contribute to improve performance.

Abstract: In this paper , we present a recursive neural network ( RNN ) model that works on a syntactic tree .

Abstract: Our model differs from previous RNN models in that the model allows for an explicit weighting of important phrases for the target task .

Abstract: We also propose to average parameters in training .

Abstract: Our experimental results on semantic relation classification show that both phrase categories and task-specific weighting significantly improve the prediction accuracy of the model .

Abstract: We also show that averaging the model parameters is effective in stabilizing the learning and improves generalization capacity .

Abstract: The proposed model marks scores competitive with state-of-the-art RNN-based models .

Recursive Neural Network ( RNN ) models are promising deep learning models which have been applied to a variety of natural language processing ( NLP ) tasks , such as sentiment classification , compound similarity , relation classification and syntactic parsing ( Hermann and Blunsom , 2013 ; Socher et al . , 2012 ; Socher et al . , 2013 ) .

RNN models can represent phrases of arbitrary length in a vector space of a fixed dimension .

Most of them use minimal syntactic information ( Socher et al . , 2012 ) .

Recently , Hermann and Blunsom ( 2013 ) proposed a method for leveraging syntactic information , namely CCG combinatory operators , to guide composition of phrases in RNN models .

While their models were successfully applied to binary sentiment classification and compound similarity tasks , there are questions yet to be answered , e.g. , whether such enhancement is beneficial in other NLP tasks as well , and whether a similar improvement can be achieved by using syntactic information of more commonly available types such as phrase categories and syntactic heads .

In this paper , we present a supervised RNN model for a semantic relation classification task .

Our model is different from existing RNN models in that important phrases can be explicitly weighted for the task .

Syntactic information used in our model includes part-of-speech ( POS ) tags , phrase categories and syntactic heads .

POS tags are used to assign vector representations to word-POS pairs .

Phrase categories are used to determine which weight matrices are chosen to combine phrases .

Syntactic heads are used to determine which phrase is weighted during combining phrases .

To incorporate task-specific information , phrases on the path between entity pairs are further weighted .

The second contribution of our work is the introduction of parameter averaging into RNN models .

In our preliminary experiments , we observed that the prediction performance of the model often fluctuates significantly between training iterations .

This fluctuation not only leads to unstable performance of the resulting models , but also makes it difficult to fine-tune the hyperparameters of the model .

Inspired by Swersky et al . ( 2010 ) , we propose to average the model parameters in the course of training .

A recent technique for deep learning models of similar vein is dropout ( Hinton et al . , 2012 ) , but averaging is simpler to implement .

Our experimental results show that our model per-Figure 1 : A recursive representations of a phrase " a word vector " with POS tags of the words ( DT , NN and NN respectively ) .

For example , the two word-POS pairs " word NN " and " vector NN " with a syntactic category N are combined to represent the phrase " word vector " .

By averaging the model parameters , our model achieves performance competitive with the MV-RNN model in Socher et al . ( 2012 ) , without using computationally expensive word-dependent matrices .

We have presented an averaged RNN model for semantic relation classification .

Our experimental results show that syntactic information such as phrase categories and heads improves the performance , and the task-specific weighting is also beneficial .

The results also demonstrate that averaging the model parameters not only stabilizes the learning but also improves the generalization capacity of the model .

As future work , we plan to combine deep learning models with richer information such as predicateargument structures .


Existing models only use pair-based features but not context for pronoun resolution regardless of its utility.

They propose a supervised approach that uses coreferential information of candidates from the local context.

The proposed method improves over the baselines both in ideal and practical setups.

Abstract: Coreferential information of a candidate , such as the properties of its antecedents , is important for pronoun resolution because it reflects the salience of the candidate in the local discourse .

Abstract: Such information , however , is usually ignored in previous learning-based systems .

Abstract: In this paper we present a trainable model which incorporates coreferential information of candidates into pronoun resolution .

Abstract: Preliminary experiments show that our model will boost the resolution performance given the right antecedents of the candidates .

Abstract: We further discuss how to apply our model in real resolution where the antecedents of the candidate are found by a separate noun phrase resolution module .

Abstract: The experimental results show that our model still achieves better performance than the baseline .

In recent years , supervised machine learning approaches have been widely explored in reference resolution and achieved considerable success ( Ge et al . , 1998 ; Soon et al . , 2001 ; Ng and Cardie , 2002 ; Strube and Muller , 2003 ; Yang et al . , 2003 ) .

Most learning-based pronoun resolution systems determine the reference relationship between an anaphor and its antecedent candidate only from the properties of the pair .

The knowledge about the context of anaphor and antecedent is nevertheless ignored .

However , research in centering theory ( Sidner , 1981 ; Grosz et al . , 1983 ; Grosz et al . , 1995 ; Tetreault , 2001 ) has revealed that the local focusing ( or centering ) also has a great effect on the processing of pronominal expressions .

The choices of the antecedents of pronouns usually depend on the center of attention throughout the local discourse segment ( Mitkov , 1999 ) .

To determine the salience of a candidate in the local context , we may need to check the coreferential information of the candidate , such as the existence and properties of its antecedents .

In fact , such information has been used for pronoun resolution in many heuristicbased systems .

The S-List model ( Strube , 1998 ) , for example , assumes that a co-referring candidate is a hearer-old discourse entity and is preferred to other hearer-new candidates .

In the algorithms based on the centering theory ( Brennan et al . , 1987 ; Grosz et al . , 1995 ) , if a candidate and its antecedent are the backwardlooking centers of two subsequent utterances respectively , the candidate would be the most preferred since the CONTINUE transition is always ranked higher than SHIFT or RETAIN .

In this paper , we present a supervised learning-based pronoun resolution system which incorporates coreferential information of candidates in a trainable model .

For each candidate , we take into consideration the properties of its antecedents in terms of features ( henceforth backward features ) , and use the supervised learning method to explore their influences on pronoun resolution .

In the study , we start our exploration on the capability of the model by applying it in an ideal environment where the antecedents of the candidates are correctly identified and the backward features are optimally set .

The experiments on MUC-6 ( 1995 ) and MUC-7 ( 1998 ) corpora show that incorporating coreferential information of candidates boosts the system performance significantly .

Further , we apply our model in the real resolution where the antecedents of the candidates are provided by separate noun phrase resolution modules .

The experimental results show that our model still outperforms the baseline , even with the low recall of the non-pronoun resolution module .

The remaining of this paper is organized as follows .

Section 2 discusses the importance of the coreferential information for candidate evaluation .

Section 3 introduces the baseline learning framework .

Section 4 presents and evaluates the learning model which uses backward fea-tures to capture coreferential information , while Section 5 proposes how to apply the model in real resolution .

Section 6 describes related research work .

Finally , conclusion is given in Section 7 .

In this paper we have proposed a model which incorporates coreferential information of candi-dates to improve pronoun resolution .

When evaluating a candidate , the model considers its adjacent antecedent by describing its properties in terms of backward features .

We first examined the effectiveness of the model by applying it in an optimal environment where the closest antecedent of a candidate is obtained correctly .

The experiments show that it boosts the success rate of the baseline system for both MUC-6 ( 4.7 % ) and MUC-7 ( 3.5 % ) .

Then we proposed how to apply our model in the real resolution where the antecedent of a non-pronoun is found by an additional non-pronoun resolution module .

Our model can still produce Success improvement ( 4.7 % for MUC-6 and 1.8 % for MUC-7 ) against the baseline system , despite the low recall of the non-pronoun resolution module .

In the current work we restrict our study only to pronoun resolution .

In fact , the coreferential information of candidates is expected to be also helpful for non-pronoun resolution .

We would like to investigate the influence of the coreferential factors on general NP reference resolution in our future work .