Although there are some open questions such as how helpful are linguistic analysis and pattern learning, the definitional question is under explored.

They evaluate a hybrid approach for Chinese definitional question answering by combining deep linguistic analysis with surface pattern learning.

They report findings such as linguistic analysis can complement pattern learning, and that a small amount of annotation is sufficient for a biographical question answering.

Abstract: We explore a hybrid approach for Chinese definitional question answering by combining deep linguistic analysis with surface pattern learning .

Abstract: We answer four questions in this study : 1 ) How helpful are linguistic analysis and pattern learning ?

Abstract: 2 ) What kind of questions can be answered by pattern matching ?

Abstract: 3 ) How much annotation is required for a pattern-based system to achieve good performance ?

Abstract: 4 ) What linguistic features are most useful ?

Abstract: Extensive experiments are conducted on biographical questions and other definitional questions .

Abstract: Major findings include : 1 ) linguistic analysis and pattern learning are complementary ; both are required to make a good definitional QA system ; 2 ) pattern matching is very effective in answering biographical questions while less effective for other definitional questions ; 3 ) only a small amount of annotation is required for a pattern learning system to achieve good performance on biographical questions ; 4 ) the most useful linguistic features are copulas and appositives ; relations also play an important role ; only some propositions convey vital facts .

Due to the ever increasing large amounts of online textual data , learning from textual data is becoming more and more important .

Traditional document retrieval systems return a set of relevant documents and leave the users to locate the specific information they are interested in .

Question answering , which combines traditional document retrieval and information extraction , solves this problem directly by returning users the specific answers .

Research in textual question answering has made substantial advances in the past few years ( Voorhees , 2004 ) .

Most question answering research has been focusing on factoid questions where the goal is to return a list of facts about a concept .

Definitional questions , however , remain largely unexplored .

Definitional questions differ from factoid questions in that the goal is to return the relevant " answer nuggets " of information about a query .

Identifying such answer nuggets requires more advanced language processing techniques .

Definitional QA systems are not only interesting as a research challenge .

They also have the potential to be a valuable complement to static knowledge sources like encyclopedias .

This is because they create definitions dynamically , and thus answer definitional questions about terms which are new or emerging ( Blair-Goldensoha et al . , 2004 ) .

One success in factoid question answering is pattern based systems , either manually constructed ( Soubbotin and Soubbotin , 2002 ) or machine learned ( Cui et al . , 2004 ) .

However , it is unknown whether such pure pattern based systems work well on definitional questions where answers are more diverse .

Deep linguistic analysis has been found useful in factoid question answering ( Moldovan et al . , 2002 ) and has been used for definitional questions ( Xu et al . , 2004 ; Harabagiu et al . , 2003 ) .

Linguistic analy-sis is useful because full parsing captures long distance dependencies between the answers and the query terms , and provides more information for inference .

However , merely linguistic analysis may not be enough .

First , current state of the art linguistic analysis such as parsing , co-reference , and relation extraction is still far below human performance .

Errors made in this stage will propagate and lower system accuracy .

Second , answers to some types of definitional questions may have strong local dependencies that can be better captured by surface patterns .

Thus we believe that combining linguistic analysis and pattern learning would be complementary and be beneficial to the whole system .

Work in combining linguistic analysis with patterns include Weischedel et al . ( 2004 ) and Jijkoun et al . ( 2004 ) where manually constructed patterns are used to augment linguistic features .

However , manual pattern construction critically depends on the domain knowledge of the pattern designer and often has low coverage ( Jijkoun et al . , 2004 ) .

Automatic pattern derivation is more appealing ( Ravichandran and Hovy , 2002 ) .

In this work , we explore a hybrid approach to combining deep linguistic analysis with automatic pattern learning .

We are interested in answering the following four questions for Chinese definitional question answering : How helpful are linguistic analysis and pattern learning in definitional question answering ?

If pattern learning is useful , what kind of question can pattern matching answer ?

How much human annotation is required for a pattern based system to achieve reasonable performance ?

We have explored a hybrid approach for definitional question answering by combining deep linguistic analysis and surface pattern learning .

For the first time , we have answered four questions regarding Chinese definitional QA : deep linguistic analysis and automatic pattern learning are complementary and may be combined ; patterns are powerful in answering biographical questions ; only a small amount of annotation ( 2 days ) is required to obtain good performance in a biographical QA system ; copulas and appositions are the most useful linguistic features ; relation extraction also helps .

Answering " What-is " questions is more challenging than answering " Who-is " questions .

To improve the performance on " What-is " questions , we could divide " What-is " questions into finer classes such as organization , location , disease , and general substance , and process them specifically .

Our current pattern matching is based on simple POS tagging which captures only limited syntactic information .

We generalize words to their corresponding POS tags .

Another possible improvement is to generalize using automatically derived word clusters , which provide semantic information .


Because of the absence of a corpus, scheme or metrics, there is no work on topic segmentation for asynchronous multi-party conversations.

They present the first email corpus with topic annotations and evaluation metrics, then evaluate existing methods and their extensions that utilize conversation structure features.

The existing LDA-based methods with conversation structure features perform well on the new email topic segmentation corpus.

Abstract: This work concerns automatic topic segmentation of email conversations .

Abstract: We present a corpus of email threads manually annotated with topics , and evaluate annotator reliability .

Abstract: To our knowledge , this is the first such email corpus .

Abstract: We show how the existing topic segmentation models ( i.e. , Lexical Chain Segmenter ( LCSeg ) and Latent Dirichlet Allocation ( LDA ) ) which are solely based on lexical information , can be applied to emails .

Abstract: By pointing out where these methods fail and what any desired model should consider , we propose two novel extensions of the models that not only use lexical information but also exploit finer level conversation structure in a principled way .

Abstract: Empirical evaluation shows that LCSeg is a better model than LDA for segmenting an email thread into topical clusters and incorporating conversation structure into these models improves the performance significantly .

With the ever increasing popularity of emails and web technologies , it is very common for people to discuss issues , events , agendas or tasks by email .

Effective processing of the email contents can be of great strategic value .

In this paper , we study the problem of topic segmentation for emails , i.e. , grouping the sentences of an email thread into a set of coherent topical clusters .

Adapting the standard definition of topic ( Galley et al . , 2003 ) to conversations / emails , we consider a topic is something about which the participant(s ) discuss or argue or express their opinions .

For example , in the email thread shown in Figure 1 , according to the majority of our annotators , participants discuss three topics ( e.g. , ' telecon cancellation ' , ' TAG document ' , and ' responding to I18N ' ) .

Multiple topics seem to occur naturally in social interactions , whether synchronous ( e.g. , chats , meetings ) or asynchronous ( e.g. , emails , blogs ) conversations .

In multi-party chat ( Elsner and Charniak , 2008 ) report an average of 2.75 discussions active at a time .

In our email corpus , we found an average of 2.5 topics per thread .

Topic segmentation is often considered a prerequisite for other higher-level conversation analysis and applications of the extracted structure are broad , encompassing : summarization ( Harabagiu and Lacatusu , 2005 ) , information extraction and ordering ( Allan , 2002 ) , information retrieval ( Dias et al . , 2007 ) , and intelligent user interfaces ( Dredze et al . , 2008 ) .

While extensive research has been conducted in topic segmentation for monologues ( e.g. , ( Malioutov and Barzilay , 2006 ) , ( Choi et al . , 2001 ) ) and synchronous dialogs ( e.g. , ( Galley et al . , 2003 ) , ( Hsueh et al . , 2006 ) ) , none has studied the problem of segmenting asynchronous multi-party conversations ( e.g. , email ) .

Therefore , there is no reliable annotation scheme , no standard corpus , and no agreedupon metrics available .

Also , it is our key hypothesis that , because of its asynchronous nature , and the use of quotation ( Crystal , 2001 ) , topics in an email thread often do not change in a sequential way .

As a result , we do not expect models which have proved successful in monologue or dialog to be as effective when they are applied to email conversations .

Our contributions in this paper aim to remedy these problems .

First , we present an email corpus annotated with topics and evaluate annotator agreement .

Second , we adopt a set of metrics to measure the local and global structural similarity between two annotations from the work on multi-party chat disentanglement ( Elsner and Charniak , 2008 ) .

Third , we show how the two state-of-the-art topic segmentation methods ( i.e. , LCSeg and LDA ) which are solely based on lexical information and make strong assumptions on the resulting topic models , can be effectively applied to emails , by having them to consider , in a principled way , a finer level structure of the underlying conversations .

Experimental results show that both LCSeg and LDA benefit when they are extended to consider the conversational structure .

When comparing the two methods , we found that LCSeg is better than LDA and this advantage is preserved when they are extended to incorporate conversational structure .

In this paper we presented an email corpus annotated for topic segmentation .

We extended LDA and LC-Seg models by incorporating the fragment quotation graph , a fine-grain model of the conversation , which is based on the analysis of quotations .

Empirical evaluation shows that the fragment quotation graph helps both these models to perform significantly better than their basic versions , with LCSeg+FQG being the best performer .


Existing distant supervision-based knowledge based-matching methods for the chemistry domain suffer from incomplete and noisy annotations due to the complex name structures in the domain.

They propose to use type ontology structures to generate distant labels and apply a multi-type disambiguation method to resolve the noisy annotation problem.

The proposed method improves the distant label generation for the subsequent model training and outperforms the state-of-the-art NER methods on the newly expert-labelled chemistry dataset.

Abstract: Scientific literature analysis needs fine-grained named entity recognition ( NER ) to provide a wide range of information for scientific discovery .

Abstract: For example , chemistry research needs to study dozens to hundreds of distinct , fine-grained entity types , making consistent and accurate annotation difficult even for crowds of domain experts .

Abstract: On the other hand , domain-specific ontologies and knowledge bases ( KBs ) can be easily accessed , constructed , or integrated , which makes distant supervision realistic for fine-grained chemistry NER .

Abstract: In distant supervision , training labels are generated by matching mentions in a document with the concepts in the knowledge bases ( KBs ) .

Abstract: However , this kind of KB-matching suffers from two major challenges : incomplete annotation and noisy annotation .

Abstract: We propose CHEMNER , an ontologyguided , distantly-supervised method for finegrained chemistry NER to tackle these challenges .

Abstract: It leverages the chemistry type ontology structure to generate distant labels with novel methods of flexible KB-matching and ontology-guided multi-type disambiguation .

Abstract: It significantly improves the distant label generation for the subsequent sequence labeling model training .

Abstract: We also provide an expertlabeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Abstract: Experimental results show that CHEMNER is highly effective , outperforming substantially the stateof-the-art NER methods ( with .25 absolute F1 score improvement ) .

Named entity recognition ( NER ) is a fundamental step in scientific literature analysis to build AI-driven systems for molecular discovery , synthetic strategy designing , and manufacturing ( Xie et al . , 2013 ; Szklarczyk et al . , 2015 ; Huang et al . , 2015 ; Szklarczyk et al . , 2017 ; de Almeida et al . , 2019 ) .

It aims to locate and classify entity mentions ( e.g. , " Suzuki-Miyaura cross-coupling reactions " ) from unstructured text into pre-defined categories ( e.g. , " coupling reactions " ) .

In the chemistry domain , previous NER studies are mostly focused on one coarse-grained entity type ( i.e. , chemicals ) ( Krallinger et al . , 2015 ; He et al . , 2020 ; Watanabe et al . , 2019 ) and rely on large amounts of manuallyannotated data for training deep learning models ( Chiu and Nichols , 2016 ; Ma and Hovy , 2016 ; Lample et al . , 2016 ; Wang et al . , 2019b ; Devlin et al . , 2019 ; Liu et al . , 2019 ) .

In real-world applications , it is important to recognize chemistry entities on diverse and finegrained types ( e.g. , " inorganic phophorus compounds " , " coupling reactions " and " catalysts " ) to provide a wide range of information for scientific discovery .

It will need dozens to hundreds of distinct types , making consistent and accurate annotation difficult even for domain experts .

On the other hand , the domain-specific ontologies and knowledge bases ( KBs ) can be easily accessed , constructed , or integrated , which makes distant supervision realistic for fine-grained chemistry NER .

Still , challenges exist for correctly recognizing the entity boundaries and accurately typing entities with distant supervision .

In distant supervision , training labels are generated by matching the mentions in a document with the concepts in the knowledge bases ( KBs ) .

( 1 ) incomplete annotation where a mention in a document can be matched only partially or missed completely due to an incomplete coverage of the KBs ( Figure 1a ) , and ( 2 ) noisy annotation where a mention can be erroneously matched due to the potential matching of multiple entity types in the KBs ( Figure 1b ) .

Due to the complex name structures ( e.g. , nested naming structures and long chemical formulas ) of chemical entities , these challenges lead to severe low-precision and low-recall for finegrained chemistry NER with distant supervision .

Several studies have attempted to address the incomplete annotation problem in distantlysupervised NER .

For example , AutoNER ( Shang et al . , 2018b ) introduces an " unknown " type that can be skipped during training to reduce the effect of false negative labeling with distant supervision .

BOND ( Liang et al . , 2020 ) leverages the power of pre-trained language models and a self-training approach to iteratively incorporate more training labels and improve the NER performance .

However , previous methods assume a high precision and reasonable coverage of KB-matching for distant label generation .

For example , the KB-matching on the CoNLL03 dataset ( Liang et al . , 2020 ) reported over 80 % on precision and over 60 % on recall .

These methods do not work well with fine-grained chemistry NER that has severe low precision and low recall with KB-matching .

Previous studies also largely ignore the noisy annotation problem by simply discarding those multi-labels during the KBmatching process ( Liang et al . , 2020 ) .

However , the noisy labels can not be simply ignored for the chemistry entities because they consist of a large portion of distant training labels .

We observe that more than 60 % of the entities have multiple labels during KB-matching in the chemistry domain .

We propose CHEMNER , an ontology-guided , distantly-supervised NER method for fine-grained chemistry NER .

Taking an input corpus , a chemistry type ontology and associated entity dictionaries collected from the KBs , we develop a novel flexible KB-matching method with TF-IDF-based majority voting to resolve the incomplete annota-tion problem .

Then we develop a novel ontologyguided multi-type disambiguation method to resolve the noisy annotation problem .

Taking the output from the above two steps as distant supervision , we further train a sequence labeling model to cover additional entities .

CHEMNER significantly improves the distant label generation for the subsequent NER model training .

We also provide an expert-labeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Experimental results show that CHEMNER is highly effective , achieving substantially better performance ( with .25 absolute F1 score improvement ) compared with the state-ofthe-art NER methods .

We have released our data and code to benefit future studies1 .

We propose CHEMNER , an ontology-guided , distantly-supervised method for fine-grained chemistry NER .

We also provide an expert labeled , chemistry NER dataset with 62 finegrained chemistry types ( e.g. , chemical compounds and chemical reactions ) .

Experimental results show that CHEMNER is highly effective , outperforming substantially the state-of-the-art NER methods on fine-grained chemistry NER .

Although achieving great performance , there is still large room for improvement of CHEMNER .

In the future , we plan to further refine and enrich the type ontology and incorporate more information in the dictionaries ( e.g. , chemical structures in the KBs ) for a better NER performance .

We also plan to apply our finegrained NER method to other scientific domains .


Existing methods guide phrase-based decoders to prefer translations which respect source-side parse trees however syntactic structures of the source do not necessarily reflect translation structure.

They propose to use binary classifiers trained on the word-aligned corpus to predict translation boundaries for source sentences and use them as soft constraints.

The proposed classifiers outperform ones with access to gold translation boundaries and a phrase-based translation model with the proposed constraints improves translation quality.

Abstract: Constrained decoding is of great importance not only for speed but also for translation quality .

Abstract: Previous efforts explore soft syntactic constraints which are based on constituent boundaries deduced from parse trees of the source language .

Abstract: We present a new framework to establish soft constraints based on a more natural alternative : translation boundary rather than constituent boundary .

Abstract: We propose simple classifiers to learn translation boundaries for any source sentences .

Abstract: The classifiers are trained directly on word-aligned corpus without using any additional resources .

Abstract: We report the accuracy of our translation boundary classifiers .

Abstract: We show that using constraints based on translation boundaries predicted by our classifiers achieves significant improvements over the baseline on large-scale Chinese-to-English translation experiments .

Abstract: The new constraints also significantly outperform constituent boundary based syntactic constrains .

It has been known that phrase-based decoding ( phrase segmentation / translation / reordering ( Chiang , 2005 ) ) should be constrained to some extent not only for transferring the NP-hard problem ( Knight , 1999 ) into a tractable one in practice but also for improving translation quality .

For example , Xiong et al . ( 2008 ) find that translation quality can be significantly improved by either prohibiting reorderings around punctuation or restricting reorderings within a 15-word window .

Recently , more linguistically motivated constraints are introduced to improve phrase-based decoding .

( Cherry , 2008 ) and ( Marton and Resnik , 2008 ) introduce syntactic constraints into the standard phrase-based decoding ( Koehn et al . , 2003 ) and hierarchical phrase-based decoding ( Chiang , 2005 ) respectively by using a counting feature which accumulates whenever hypotheses violate syntactic boundaries of source-side parse trees .

( Xiong et al . , 2009 ) further presents a bracketing model to include thousands of context-sensitive syntactic constraints .

All of these approaches achieve their improvements by guiding the phrase-based decoder to prefer translations which respect source-side parse trees .

One major problem with such constituent boundary based constraints is that syntactic structures of the source language do not necessarily reflect translation structures where the source and target language correspond to each other .

In this paper , we investigate building classifiers that directly address the problem of translation boundary , rather than extracting constituent boundary from sourceside parsers built for a different purpose .

A translation boundary is a position in the source sequence which begins or ends a translation zone1 spanning multiple source words .

In a translation zone , the source phrase is translated as a unit .

Reorderings which cross translation zones are not desirable .

Inspired by ( Roark and Hollingshead , 2008 ) which introduces classifiers to decide if a word can begin / end a multi-word constituent , we build two discriminative classifiers to tag each word in the source sequence with a binary class label .

The first classifier decides if a word can begin a multi-sourceword translation zone ; the second classifier decides if a word can end a multi-source-word translation zone .

Given a partial translation covering source sequence ( i , j ) with start word c i and end word c j2 , this translation can be penalized if the first classifier decides that the start word c i can not be a beginning translation boundary or the second classifier decides that the end word c j can not be an ending translation boundary .

In such a way , we can guide the decoder to boost hypotheses that respect translation boundaries and therefore the common translation structure shared by the source and target language , rather than the syntactic structure of the source language .

We report the accuracy of such classifiers by comparing their outputs with " gold " translation boundaries obtained from reference translations on the development set .

We integrate translation boundary based constraints into phrase-based decoding and display that they improve translation quality significantly in large-scale experiments .

Furthermore , we confirm that they also significantly outperform constituent boundary based syntactic constraints .

In this paper , we have presented a simple approach to learn translation boundaries on source sentences .

The learned translation boundaries are used to constrain phrase-based decoding in a soft manner .

The whole approach has several properties .

• First , it is based on a simple classification task that can achieve considerably high accuracy when taking translation divergences into account using simple models and features .

• Second , the classifier output can be straightforwardly used to constrain phrase-based decoder .

• Finally , we have empirically shown that , to build soft constraints for phrase-based decoding , translation boundary predicted by our classifier is a better choice than constituent boundary deduced from source-side parse tree .

Future work in this direction will involve trying different methods to define more informative translation boundaries , such as a boundary to begin / end a swapping .

We would also like to investigate new methods to incorporate automatically learned translation boundaries more efficiently into decoding in an attempt to further improve search in both speed and accuracy .


Information extraction systems cannot extract role filters which are arguments to verbs but are not event-specific because they do not recognize the contexts as relevant.

They propose a unified probabilistic model composed of sentential event recognition and plausible role filter recognition models which run both on phrasal and sentence levels.

The proposed unified model outperforms the existing systems that rely on the local phrasal context of two information extraction datasets.

Abstract: Information Extraction ( IE ) systems that extract role fillers for events typically look at the local context surrounding a phrase when deciding whether to extract it .

Abstract: Often , however , role fillers occur in clauses that are not directly linked to an event word .

Abstract: We present a new model for event extraction that jointly considers both the local context around a phrase along with the wider sentential context in a probabilistic framework .

Abstract: Our approach uses a sentential event recognizer and a plausible role-filler recognizer that is conditioned on event sentences .

Abstract: We evaluate our system on two IE data sets and show that our model performs well in comparison to existing IE systems that rely on local phrasal context .

Information Extraction ( IE ) systems typically use extraction patterns ( e.g. , Soderland et al . ( 1995 ) , Riloff ( 1996 ) , Yangarber et al . ( 2000 ) , Califf and Mooney ( 2003 ) ) or classifiers ( e.g. , Freitag ( 1998 ) , Freitag and McCallum ( 2000 ) , Chieu et al . ( 2003 ) , Bunescu and Mooney ( 2004 ) ) to extract role fillers for events .

Most IE systems consider only the immediate context surrounding a phrase when deciding whether to extract it .

For tasks such as named entity recognition , immediate context is usually sufficient .

But for more complex tasks , such as event extraction , a larger field of view is often needed to understand how facts tie together .

Most IE systems are designed to identify role fillers that appear as arguments to event verbs or nouns , either explicitly via syntactic relations or implicitly via proximity ( e.g. , John murdered Tom or the murder of Tom by John ) .

But many facts are presented in clauses that do not contain event words , requiring discourse relations or deep structural analysis to associate the facts with event roles .

For example , consider the sentences below :

and 30 were injured in India after terrorists launched an attack on the Taj Hotel .

in Mexico City and its surrounding suburbs in a Swine Flu outbreak .

after a tractor-trailer collided with a bus in Arkansas .

in Baghdad last night in a resurgence of bomb attacks in the capital city .

and $ 50 million in damage was caused by a hurricane that hit Miami on Friday .

to make way for modern , safer bridges that will be constructed early next year .

These examples illustrate a common phenomenon in text where information is not explicitly stated as filling an event role , but readers have no trouble making this inference .

The role fillers above ( seven people , two bridges ) occur as arguments to verbs that reveal state information ( death , destruction ) but are not event-specific ( i.e. , death and destruction can result from a wide variety of incident types ) .

IE systems often fail to extract these role fillers because these systems do not recognize the immediate context as being relevant to the specific type of event that they are looking for .

We propose a new model for information extraction that incorporates both phrasal and sentential evidence in a unified framework .

Our unified probabilistic model , called GLACIER , consists of two components : a model for sentential event recognition and a model for recognizing plausible role fillers .

The Sentential Event Recognizer offers a probabilistic assessment of whether a sentence is discussing a domain-relevant event .

The Plausible Role-Filler Recognizer is then conditioned to identify phrases as role fillers based upon the assumption that the surrounding context is discussing a relevant event .

This unified probabilistic model allows the two components to jointly make decisions based upon both the local evidence surrounding each phrase and the " peripheral vision " afforded by the sentential event recognizer .

Section 2 positions our research with respect to related work .

Section 3 presents our unified probabilistic model for information extraction .

Section 4 shows experimental results on two IE data sets , and Section 5 discusses directions for future work .

We presented a unified model for IE that balances the influence of sentential context with local contextual evidence to improve the performance of event-based IE .

Our experimental results showed that using sentential contexts indeed produced better results on two IE data sets .

Our current model uses supervised learning , so one direction for future work is to adapt the model for weakly supervised learning .

We also plan to incorporate discourse features and investigate even wider contexts to capture broader discourse effects .


For evaluating machine translation systems, manual evaluation is costly and inconsistent and the automatic counterpart is unreliable because it requires references.

They propose a new evaluation framework which assesses adequacy and fluency by using a cross-language latent indexing approach and n-gram based language modelling without references.

The proposed method underperforms two standard metrics on a large collection in different languages and shows however it does not require reference translations.

Abstract: This work introduces AM-FM , a semantic framework for machine translation evaluation .

Abstract: Based upon this framework , a new evaluation metric , which is able to operate without the need for reference translations , is implemented and evaluated .

Abstract: The metric is based on the concepts of adequacy and fluency , which are independently assessed by using a cross-language latent semantic indexing approach and an n-gram based language model approach , respectively .

Abstract: Comparative analyses with conventional evaluation metrics are conducted on two different evaluation tasks ( overall quality assessment and comparative ranking ) over a large collection of human evaluations involving five European languages .

Abstract: Finally , the main pros and cons of the proposed framework are discussed along with future research directions .

Evaluation has always been one of the major issues in Machine Translation research , as both human and automatic evaluation methods exhibit very important limitations .

On the one hand , although highly reliable , in addition to being expensive and time consuming , human evaluation suffers from inconsistency problems due to inter-and intraannotator agreement issues .

On the other hand , while being consistent , fast and cheap , automatic evaluation has the major disadvantage of requiring reference translations .

This makes automatic evaluation not reliable in the sense that good translations not matching the available references are evaluated as poor or bad translations .

The main objective of this work is to propose and evaluate AM-FM , a semantic framework for assessing translation quality without the need for reference translations .

The proposed framework is theoretically grounded on the classical concepts of adequacy and fluency , and it is designed to account for these two components of translation quality in an independent manner .

First , a cross-language latent semantic indexing model is used for assessing the adequacy component by directly comparing the output translation with the input sentence it was generated from .

Second , an n-gram based language model of the target language is used for assessing the fluency component .

Both components of the metric are evaluated at the sentence level , providing the means for defining and implementing a sentence-based evaluation metric .

Finally , the two components are combined into a single measure by implementing a weighted harmonic mean , for which the weighting factor can be adjusted for optimizing the metric performance .

The rest of the paper is organized as follows .

Section 2 , presents some background work and the specific dataset that has been used in the experimental work .

Section 3 , provides details on the proposed AM-FM framework and the specific metric implementation .

Section 4 presents the results of the conducted comparative evaluations .

Finally , section 5 presents the main conclusions and relevant issues to be dealt with in future research .

This work presented AM-FM , a semantic framework for translation quality assessment .

Two comparative evaluations with standard metrics have been conducted over a large collection of humangenerated scores involving different languages .

Although the obtained performance is below standard metrics , the proposed method has the main advantage of not requiring reference translations .

Notice that a monolingual version of AM-FM is also possible by using monolingual latent semantic indexing ( Landauer et al . , 1998 ) along with a set of reference translations .

A detailed evaluation of a monolingual implementation of AM-FM can be found in Banchs and Li ( 2011 ) .

As future research , we plan to study the impact of different dataset sizes and vector space model parameters for improving the performance of the AM component of the metric .

This will include the study of learning curves based on the amount of training data used , and the evaluation of different vector model construction strategies , such as removing stop-words and considering bigrams and word categories in addition to individual words .

Finally , we also plan to study alternative uses of AM-FM within the context of statistical machine translation as , for example , a metric for MERT optimization , or using the AM component alone as an additional feature for decoding , rescoring and/or confidence estimation .


Existing methods for Video Question Answering lack temporal modelling, reasoning capability, or an ability to learn dependencies between distant features.

They propose a motion module and an appearance module which embeds cross-modal representations and a motion-appearance fusion module to integrate them based on question features.

The proposed method achieves the new state-of-the-art performance on TGIF-QA and MSVD-QA datasets and they also present qualitative analysis with visualizations.

Abstract: Video Question Answering is a task which requires an AI agent to answer questions grounded in video .

Abstract: This task entails three key challenges : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , causality ) , and ( 3 ) cross-modal grounding between language and vision information .

Abstract: We propose Motion-Appearance Synergistic Networks ( MASN ) , which embed two crossmodal features grounded on motion and appearance information and selectively utilize them depending on the question 's intentions .

Abstract: MASN consists of a motion module , an appearance module , and a motion-appearance fusion module .

Abstract: The motion module computes the action-oriented cross-modal joint representations , while the appearance module focuses on the appearance aspect of the input video .

Abstract: Finally , the motion-appearance fusion module takes each output of the motion module and the appearance module as input , and performs question-guided fusion .

Abstract: As a result , MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets .

Abstract: We also conduct qualitative analysis by visualizing the inference results of MASN .

Abstract: The code is available at https://github.com/ ahjeongseo / MASN-pytorch .

Recently , research in natural language processing and computer vision has made significant progress in artificial intelligence ( AI ) .

Thanks to this , visionlanguage tasks such as image captioning ( Xu et al . , 2015 ) , visual question answering ( VQA ) ( Antol et al . , 2015 ; Goyal et al . , 2017 ) , and visual commonsense reasoning ( VCR ) ( Zellers et al . , 2019 ) have been introduced to the research community , along with some benchmark datasets .

In particular , video question answering ( video QA ) tasks ( Xu et al . , 2016 ; Jang et al . , 2017 ; Lei et al . , 2018 ;

Yu et al . , 2019 ; Choi et al . , 2020 ) have been proposed with the goal of reasoning over higher-level visionlanguage interactions .

In contrast to QA tasks based on static images , the questions presented in the video QA dataset vary from frame-level questions regarding the appearance of objects ( e.g. , what is the color of the hat ? )

to questions regarding action and causality ( e.g. , what does the man do after opening a door ? ) .

There are three crucial challenges in video QA : ( 1 ) understand the intention of various questions , ( 2 ) capturing various elements of the input video ( e.g. , object , action , and causality ) , and ( 3 ) crossmodal grounding between language and vision information .

To tackle these challenges , previous studies ( Li et al . , 2019 ; Jiang et al . , 2020 ; Huang et al . , 2020 ) have mainly explored this task by jointly embedding the features from the pre-trained word embedding model ( Pennington et al . , 2014 ) and the object detection models ( He et al . , 2016 ; Ren et al . , 2016 ) .

However , as discussed in ( Gao et al . , 2018 ) , the use of the visual features extracted from the object detection models suffers from motion analysis since the object detection model lacks temporal modeling .

To enforce the motion analysis , a few approaches ( Xu et al . , 2017 ; Gao et al . , 2018 ) have employed additional visual features ( Tran et al . , 2015 ) ( i.e. , motion features ) which were widely used in the action recognition domain , but their reasoning capability is still limited .

They typically employed recurrent models ( e.g. , LSTM ) to embed a long sequence of the visual features .

Due to the problem of long-term dependency in recurrent models ( Bengio et al . , 1993 ) , their proposed methods may fail to learn dependencies between distant features .

In this paper , we propose Motion-Appearance The output from the fusion module is used to derive answers .

For question features , the word-level representation F Q is integrated with the visual features in the VQ interaction submodule .

The last hidden units q from the bi-LSTM are used to combine appearance and motion features .

Synergistic Networks ( MASN ) for video question answering which consist of three kinds of modules : the motion module , the appearance module , and the motion-appearance fusion module .

As shown in Figure 1 , the motion module and the appearance module aim to embed rich cross-modal representations .

These two modules have the same architecture except that the motion module takes the motion features extracted from I3D as visual features and the appearance module utilizes the appearance features extracted from ResNet .

Each of these modules first constructs the object graphs via graph convolutional networks ( GCN ) to compute the relationships among objects in each visual feature .

Then , the vision-question interaction module performs cross-modal grounding between the output of the GCNs and the question features .

The motion module and the appearance module each yield cross-modal representations of the motion and the appearance aspects of the input video respectively .

The motion-appearance fusion module finally integrates these two features based on the question features .

The main contributions of our paper are as follows .

First , we propose Motion-Appearance Synergistic Networks ( MASN ) for video question answering based on three modules , the motion module , the appearance module , and the motionappearance fusion module .

Second , we validate MASN on the large-scale video question answering datasets TGIF-QA , MSVD-QA , and MSRVTT-QA .

MASN achieves the new state-of-the-art performance on TGIF-QA and MSVD-QA .

We perform ablation studies to validate the effectiveness of our proposed methods .

Finally , we conduct a qualitative analysis of MASN by visualizing inference results .

In this paper , we proposed a Motion-Appearance Synergistic Networks to fuse and create a synergy between motion and appearance features .

Through the Motion and Appearance modules , MASN manages to find motion and appearance clues to solve the question , while modulating the amount of information used of each type through the Fusion module .

Experimental results on three benchmark datasets show the effectiveness of our proposed MASN architecture compared to other models .


Semantic relation detection is useful for information retrieval and question answering however adapting the knowledge extracted for new semantic relations remains as a challenge.

They propose to create detectors for new relations from existing relations by projecting new instances onto a lower topic space constructed from existing relation extractors.

The proposed background-knowledge-based features generated from the Wikipedia relation repository outperform the state-of-the-art approaches on Wikipedia and ACE data.

Abstract: This paper describes a novel approach to the semantic relation detection problem .

Abstract: Instead of relying only on the training instances for a new relation , we leverage the knowledge learned from previously trained relation detectors .

Abstract: Specifically , we detect a new semantic relation by projecting the new relation 's training instances onto a lower dimension topic space constructed from existing relation detectors through a three step process .

Abstract: First , we construct a large relation repository of more than 7,000 relations from Wikipedia .

Abstract: Second , we construct a set of non-redundant relation topics defined at multiple scales from the relation repository to characterize the existing relations .

Abstract: Similar to the topics defined over words , each relation topic is an interpretable multinomial distribution over the existing relations .

Abstract: Third , we integrate the relation topics in a kernel function , and use it together with SVM to construct detectors for new relations .

Abstract: The experimental results on Wikipedia and ACE data have confirmed that backgroundknowledge-based topics generated from the Wikipedia relation repository can significantly improve the performance over the state-of-theart relation detection approaches .

Detecting semantic relations in text is very useful in both information retrieval and question answering because it enables knowledge bases to be leveraged to score passages and retrieve candidate answers .

To extract semantic relations from text , three types of approaches have been applied .

Rule-based methods ( Miller et al . , 2000 ) employ a number of linguistic rules to capture relation patterns .

Featurebased methods ( Kambhatla , 2004 ; Zhao and Grishman , 2005 ) transform relation instances into a large amount of linguistic features like lexical , syntactic and semantic features , and capture the similarity between these feature vectors .

Recent results mainly rely on kernel-based approaches .

Many of them focus on using tree kernels to learn parse tree structure related features ( Collins and Duffy , 2001 ; Culotta and Sorensen , 2004 ; Bunescu and Mooney , 2005 ) .

Other researchers study how different approaches can be combined to improve the extraction performance .

For example , by combining tree kernels and convolution string kernels , ( Zhang et al . , 2006 ) achieved the state of the art performance on ACE ( ACE , 2004 ) , which is a benchmark dataset for relation extraction .

Although a large set of relations have been identified , adapting the knowledge extracted from these relations for new semantic relations is still a challenging task .

Most of the work on domain adaptation of relation detection has focused on how to create detectors from ground up with as little training data as possible through techniques such as bootstrapping ( Etzioni et al . , 2005 ) .

We take a different approach , focusing on how the knowledge extracted from the existing relations can be reused to help build detectors for new relations .

We believe by reusing knowledge one can build a more cost effective relation detector , but there are several challenges associated with reusing knowledge .

The first challenge to address in this approach is how to construct a relation repository that has suffi-cient coverage .

In this paper , we introduce a method that automatically extracts the knowledge characterizing more than 7,000 relations from Wikipedia .

Wikipedia is comprehensive , containing a diverse body of content with significant depth and grows rapidly .

Wikipedia 's infoboxes are particularly interesting for relation extraction .

They are short , manually-created , and often have a relational summary of an article : a set of attribute / value pairs describing the article 's subject .

Another challenge is how to deal with overlap of relations in the repository .

For example , Wikipedia authors may make up a name when a new relation is needed without checking if a similar relation has already been created .

We refine the relation repository based on an unsupervised multiscale analysis of the correlations between existing relations .

This method is parameter free , and able to produce a set of non-redundant relation topics defined at multiple scales .

Similar to the topics defined over words ( Blei et al . , 2003 ) , we define relation topics as multinomial distributions over the existing relations .

The relation topics extracted in our approach are interpretable , orthonormal to each other , and can be used as basis relations to re-represent the new relation instances .

The third challenge is how to use the relation topics for a relation detector .

We map relation instances in the new domains to the relation topic space , resulting in a set of new features characterizing the relationship between the relation instances and existing relations .

By doing so , background knowledge from the existing relations can be introduced into the new relations , which overcomes the limitations of the existing approaches when the training data is not sufficient .

Our work fits in to a class of relation extraction research based on " distant supervision " , which studies how knowledge and resources external to the target domain can be used to improve relation extraction .

( Mintz et al . , 2009 ; Jiang , 2009 ; Chan and Roth , 2010 ) .

One distinction between our approach and other existing approaches is that we represent the knowledge from distant supervision using automatically constructed topics .

When we test on new instances , we do not need to search against the knowledge base .

In addition , our topics also model the indirect relationship between relations .

Such information can not be directly found from the knowledge base .

The contributions of this paper are three-fold .

Firstly , we extract a large amount of training data for more than 7,000 semantic relations from Wikipedia ( Wikipedia , 2011 ) and DBpedia ( Auer et al . , 2007 ) .

A key part of this step is how we handle noisy data with little human effort .

Secondly , we present an unsupervised way to construct a set of relation topics at multiple scales .

This step is parameter free , and results in a nonredundant , multiscale relation topic space .

Thirdly , we design a new kernel for relation detection by integrating the relation topics into the relation detector construction .

The experimental results on Wikipedia and ACE data ( ACE , 2004 ) have confirmed that background-knowledge-based features generated from the Wikipedia relation repository can significantly improve the performance over the state-of-the-art relation detection approaches .

This paper proposes a novel approach to create detectors for new relations integrating the knowledge extracted from the existing relations .

Firstly , we pro- vide an automatic way to collect training data for more than 7,000 relations from Wikipedia and DBpedia .

Different from the topics defined over words , relation topics are defined over the existing relations .

Thirdly , we design a new kernel for relation detection by integrating the relation topics in the representation of the relation instances .

By leveraging the knowledge extracted from the Wikipedia relation repository , our approach significantly improves the performance over the state-of-the-art approaches on ACE data .

This paper makes use of all DBpedia relations to create relation topics .

It is possible that using a subset of them ( more related to the target relations ) might improve the performance .