Recommending interesting documents to users is a fundamental feature of many systems, however; current systems do not directly model it but relevance.

They propose a CNN-based model that models documents as a sequence of words in latent space and trains it on millions of Web transitions.

They show that their model outperforms classic/recent models on two tasks that evaluate the interestingness of documents significantly.

Abstract: This paper presents a deep semantic similarity model ( DSSM ) , a special type of deep neural networks designed for text analysis , for recommending target documents to be of interest to a user based on a source document that she is reading .

Abstract: We observe , identify , and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents , which we collect from commercial Web browser logs .

Abstract: The DSSM is trained on millions of Web transitions , and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized .

Abstract: The effectiveness of the DSSM is demonstrated using two interestingness tasks : automatic highlighting and contextual entity search .

Abstract: The results on large-scale , real-world datasets show that the semantics of documents are important for modeling interestingness and that the DSSM leads to significant quality improvement on both tasks , outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models .

Tasks of predicting what interests a user based on the document she is reading are fundamental to many online recommendation systems .

A recent survey is due to Ricci et al . ( 2011 ) .

In this paper , we exploit the use of a deep semantic model for two such interestingness tasks in which document semantics play a crucial role : automatic highlighting and contextual entity search .

In this task we want a recommendation system to automatically discover the entities ( e.g. , a person , location , organi-zation etc . ) that interest a user when reading a document and to highlight the corresponding text spans , referred to as keywords afterwards .

We show in this study that document semantics are among the most important factors that influence what is perceived as interesting to the user .

For example , we observe in Web browsing logs that when a user reads an article about a movie , she is more likely to browse to an article about an actor or character than to another movie or the director .

After identifying the keywords that represent the entities of interest to the user , we also want the system to recommend new , interesting documents by searching the Web for supplementary information about these entities .

The task is challenging because the same keywords often refer to different entities , and interesting supplementary information to the highlighted entity is highly sensitive to the semantic context .

For example , " Paul Simon " can refer to many people , such as the singer and the senator .

Consider an article about the music of Paul Simon and another about his life .

Related content about his upcoming concert tour is much more interesting in the first context , while an article about his family is more interesting in the second .

At the heart of these two tasks is the notion of interestingness .

In this paper , we model and make use of this notion of interestingness with a deep semantic similarity model ( DSSM ) .

The model , extending from the deep neural networks shown recently to be highly effective for speech recognition ( Hinton et al . , 2012 ; Deng et al . , 2013 ) and computer vision ( Krizhevsky et al . , 2012 ; Markoff , 2014 ) , is semantic because it maps documents to feature vectors in a latent semantic space , also known as semantic representations .

The model is deep because it employs a neural network with several hidden layers including a special convolutional-pooling structure to identify keywords and extract hidden semantic features at different levels of abstractions , layer by layer .

The semantic representation is computed through a deep neural network after its training by backpropagation with respect to an objective tailored to the respective interestingness tasks .

We obtain naturally occurring " interest " signals by observing Web browser transitions , from a source document to a target document , in Web usage logs of a commercial browser .

Our training data is sampled from these transitions .

The use of the DSSM to model interestingness is motivated by the recent success of applying related deep neural networks to computer vision ( Krizhevshy et al . 2012 ; Markoff , 2014 ) , speech recognition ( Hinton et al . 2012 ) , text processing ( Collobert et al . 2011 ) , and Web search ( Huang et al . 2013 ) .

Among them , ( Huang et al . 2013 ) is most relevant to our work .

They also use a deep neural network to map documents to feature vectors in a latent semantic space .

However , their model is designed to represent the relevance between queries and documents , which differs from the notion of interestingness between documents studied in this paper .

It is often the case that a user is interested in a document because it provides supplementary information about the entities or concepts she encounters when reading another document although the overall contents of the second documents is not highly relevant .

For example , a user may be interested in knowing more about the history of University of Washington after reading the news about President Obama 's visit to Seattle .

To better model interestingness , we extend the model of Huang et al . ( 2013 ) in two significant aspects .

First , while Huang et al . treat a document as a bag of words for semantic mapping , the DSSM treats a document as a sequence of words and tries to discover prominent keywords .

These keywords represent the entities or concepts that might interest users , via the convolutional and max-pooling layers which are related to the deep models used for computer vision ( Krizhevsky et al . , 2013 ) and speech recognition ( Deng et al . , 2013a ) but are not used in Huang et al . 's model .

The DSSM then forms the high-level semantic representation of the whole document based on these keywords .

Second , instead of directly computing the document relevance score using cosine similarity in the learned semantic space , as in Huang et al . ( 2013 ) , we feed the features derived from the semantic representations of documents to a ranker which is trained in a supervised manner .

As a result , a document that is not highly relevant to another document a user is reading ( i.e. , the distance between their derived feature 1 We stress here that , although the click signal is available to form a dataset and a gold standard ranker ( to be described in vectors is big ) may still have a high score of interestingness because the former provides useful information about an entity mentioned in the latter .

Such information and entity are encoded , respectively , by ( some subsets of ) the semantic features in their corresponding documents .

In Sections 4 and 5 , we empirically demonstrate that the aforementioned two extensions lead to significant quality improvements for the two interestingness tasks presented in this paper .

Before giving a formal description of the DSSM in Section 3 , we formally define the interestingness function , and then introduce our data set of naturally occurring interest signals .

Modeling interestingness is fundamental to many online recommendation systems .

We obtain naturally occurring interest signals by observing Web browsing transitions where users click from one webpage to another .

We propose to model this " interestingness " with a deep semantic similarity model ( DSSM ) , based on deep neural networks with special convolutional-pooling structure , mapping source-target document pairs to feature vectors in a latent semantic space .

We train the DSSM using browsing transitions between documents .

Finally , we demonstrate the effectiveness of our model on two interestingness tasks : automatic highlighting and contextual entity search .

Our results on large-scale , real-world datasets show that the semantics of documents computed by the DSSM are important for modeling interestingness and that the new model leads to significant improvements on both tasks .

DSSM is shown to outperform not only the classic document models that do not use ( latent ) semantics but also stateof-the-art topic models that do not have the deep and convolutional architecture characterizing the DSSM .

One area of future work is to extend our method to model interestingness given an entire user session , which consists of a sequence of browsing events .

We believe that the prior browsing and interaction history recorded in the session provides additional signals for predicting interestingness .

To capture such signals , our model needs to be extended to adequately represent time series ( e.g. , causal relations and consequences of actions ) .

One potentially effective model for such a purpose is based on the architecture of recurrent neural networks ( e.g. , Mikolov et al . 2010 ; Chen and Deng , 2014 ) , which can be incorporated into the deep semantic model proposed in this paper .


Current evaluation metrics for text simplification task only focuses on lexical but not on structural aspects and do not strongly correlate with human evaluation.

They propose a reference-free metric that uses semantic parsers to decompose inputs and outputs before comparing to assess their semantic structures.

Human evaluation reveals that the proposed metric correlates better with human judgements and also complements existing reference-based metrics.

Abstract: Current measures for evaluating text simplification systems focus on evaluating lexical text aspects , neglecting its structural aspects .

Abstract: In this paper we propose the first measure to address structural aspects of text simplification , called SAMSA .

Abstract: It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output .

Abstract: SAMSA provides a reference-less automatic evaluation procedure , avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence .

Abstract: Our human evaluation experiments show both SAMSA 's substantial correlation with human judgments , as well as the deficiency of existing reference-based measures in evaluating structural simplification .

Text simplification ( TS ) addresses the translation of an input sentence into one or more simpler sentences .

It is a useful preprocessing step for several NLP tasks , such as machine translation ( Chandrasekar et al . , 1996 ; Mishra et al . , 2014 ) and relation extraction ( Niklaus et al . , 2016 ) , and has also been shown useful in the development of reading aids , e.g. , for people with dyslexia ( Rello et al . , 2013 ) or non-native speakers ( Siddharthan , 2002 ) .

The task has attracted much attention in the past decade ( Zhu et al . , 2010 ; Woodsend and Lapata , 2011 ; Wubben et al . , 2012 ; Siddharthan and Angrosh , 2014 ; Narayan and Gardent , 2014 ) , but has yet to converge on an evaluation protocol that yields comparable results across different methods and strongly correlates with human judgments .

This is in part due to the difficulty to combine the effects of different simplification operations ( e.g. , deletion , splitting and substitution ) .

al . ( 2016 ) has recently made considerable progress towards that goal , and proposed to tackle it both by using an improved reference-based measure , named SARI , and by increasing the number of references .

However , their research focused on lexical , rather than structural simplification , which provides a complementary view of TS quality as this paper will show .

This paper focuses on the evaluation of the structural aspects of the task .

We introduce the semantic measure SAMSA ( Simplification Automatic evaluation Measure through Semantic Annotation ) , the first structure-aware measure for TS in general , and the first to use semantic structure in this context in particular .

SAMSA stipulates that an optimal split of the input is one where each predicate-argument structure is assigned its own sentence , and measures to what extent this assertion holds for the input-output pair in question , by using semantic structure .

SAMSA focuses on the core semantic components of the sentence , and is tolerant towards the deletion of other units .

2For example , SAMSA will assign a high score to the output split " John got home .

for the input sentence " John got home and gave Mary a call .

" , as it splits each of its predicate-argument structures to a different sentence .

Splits that alter predicate-argument relations such as " John got home and gave .

Mary called . " are penalized by SAMSA .

SAMSA 's use of semantic structures for TS evaluation has several motivations .

First , it provides means to measure the extent to which the meaning of the source is preserved in the output .

Second , it provides means for measuring whether the input sentence was split to semantic units of the right granularity .

Third , defining a semantic measure that does not require references avoids the difficulties incurred by their non-uniqueness , and the difficulty in collecting high quality references , as reported by Xu et al . ( 2015 ) and by Narayan and Gardent ( 2014 ) with respect to the Parallel Wikipedia Corpus ( PWKP ; Zhu et al . , 2010 ) .

SAMSA is further motivated by its use of semantic annotation only on the source side , which allows to evaluate multiple systems using same source-side annotation , and avoids the need to parse system outputs , which can be garbled .

In this paper we use the UCCA scheme for defining semantic structure ( Abend and Rappoport , 2013 ) .

UCCA has been shown to be preserved remarkably well across translations ( Sulem et al . , 2015 ) and has also been successfully used for machine translation evaluation ( Birch et al . , 2016 )

We note , however , that SAMSA can be adapted to work with any semantic scheme that captures predicate-argument relations , such as AMR ( Banarescu et al . , 2013 ) or Discourse Representation Structures ( Kamp , 1981 ) , as used by Narayan and Gardent ( 2014 ) .

We experiment with SAMSA both where semantic annotation is carried out manually , and where it is carried out by a parser .

We conduct human rating experiments and compare the resulting system rankings with those predicted by SAMSA .

We find that SAMSA 's rankings obtain high correlations with human rankings , and compare favorably to existing referencebased measures for TS .

Moreover , our results show that existing measures , which mainly target lexical simplification , are ill-suited to predict human judgments where structural simplification is involved .

Finally , we apply SAMSA to the dataset of the QATS shared task on simplification evaluation ( Štajner et al . , 2016 ) .

We find that SAMSA obtains comparative correlation with human judgments on the task , despite operating in a more restricted setting , as it does not use human ratings as training data and focuses only on structural aspects of simplicity .

Section 5 details the collection of human judgments .

Our experimental setup for comparing our human and automatic rankings is given in Section 6 , and results are given in Section 7 , showing superior results for SAMSA .

A discussion on the results is presented in Section 8 .

Section 9 presents experiments with SAMSA on the QATS evaluation benchmark .

We presented the first structure-aware metric for text simplification , SAMSA , and the first evaluation experiments that directly target the structural simplification component , separately from the lexical component .

We argue that the structural and lexical dimensions of simplification are loosely related , and that TS evaluation protocols should assess both .

We empirically demonstrate that strong measures that assess lexical simplification quality ( notably SARI ) , fail to correlate with human judgments when structural simplification is performed by the evaluated systems .

Our experiments show that SAMSA correlates well with human judgments in such settings , which demonstrates its usefulness for evaluating and tuning statistical simplification systems , and shows that structural evaluation provides a complementary perspective on simplification quality .


To obtain a rich representation for parsing, manually designed templates have been used which can cause a lack of clear linguistic interpretation or design difficulties.

They propose to use tensors to map high-dimensional feature vectors into low-dimensional representations which can also represent syntactic relations and use them for online training.

The proposed approach outperforms the Turbo and MST parsers on 14 languages CoNLL 2008 and 2006, and it performs especially well without part-of-speech tags.

Abstract: Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , highdimensional feature representations .

Abstract: A small subset of such features is often selected manually .

Abstract: This is problematic when features lack clear linguistic meaning as in embeddings or when the information is blended across features .

Abstract: In this paper , we use tensors to map high-dimensional feature vectors into low dimensional representations .

Abstract: We explicitly maintain the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

Abstract: Our parser consistently outperforms the Turbo and MST parsers across 14 different languages .

Abstract: We also obtain the best published UAS results on 5 languages .

Finding an expressive representation of input sentences is crucial for accurate parsing .

Syntactic relations manifest themselves in a broad range of surface indicators , ranging from morphological to lexical , including positional and part-of-speech ( POS ) tagging features .

Traditionally , parsing research has focused on modeling the direct connection between the features and the predicted syntactic relations such as head-modifier ( arc ) relations in dependency parsing .

Even in the case of firstorder parsers , this results in a high-dimensional vector representation of each arc .

Discrete features , and their cross products , can be further complemented with auxiliary information about words participating in an arc , such as continuous vector representations of words .

The exploding dimensionality of rich feature vectors must then be balanced with the difficulty of effectively learning the associated parameters from limited training data .

A predominant way to counter the high dimensionality of features is to manually design or select a meaningful set of feature templates , which are used to generate different types of features ( Mc-Donald et al . , 2005a ; Koo and Collins , 2010 ; Martins et al . , 2013 ) .

Direct manual selection may be problematic for two reasons .

First , features may lack clear linguistic interpretation as in distributional features or continuous vector embeddings of words .

Second , designing a small subset of templates ( and features ) is challenging when the relevant linguistic information is distributed across the features .

For instance , morphological properties are closely tied to part-of-speech tags , which in turn relate to positional features .

Therefore , we may suffer a performance loss if we select only a small subset of the features .

On the other hand , by including all the rich features , we face over-fitting problems .

We depart from this view and leverage highdimensional feature vectors by mapping them into low dimensional representations .

We begin by representing high-dimensional feature vectors as multi-way cross-products of smaller feature vectors that represent words and their syntactic relations ( arcs ) .

The associated parameters are viewed as a tensor ( multi-way array ) of low rank , and optimized for parsing performance .

By explicitly representing the tensor in a low-rank form , we have direct control over the effective dimensionality of the set of parameters .

We obtain role-dependent low-dimensional representations for words ( head , modifier ) that are specifically tailored for parsing accuracy , and use standard online algorithms for optimizing the low-rank tensor components .

The overall approach has clear linguistic and computational advantages :

• Our low dimensional embeddings are tailored to the syntactic context of words ( head , modifier ) .

This low dimensional syntactic abstraction can be thought of as a proxy to manually constructed POS tags .

• By automatically selecting a small number of dimensions useful for parsing , we can leverage a wide array of ( correlated ) features .

Unlike parsers such as MST , we can easily benefit from auxiliary information ( e.g. , word vectors ) appended as features .

We implement the low-rank factorization model in the context of first-and third-order dependency parsing .

The model was evaluated on 14 languages , using dependency data from CoNLL 2008 and CoNLL 2006 .

We compare our results against the MST ( McDonald et al . , 2005a ) and Turbo ( Martins et al . , 2013 ) parsers .

The low-rank parser achieves average performance of 89.08 % across 14 languages , compared to 88.73 % for the Turbo parser , and 87.19 % for MST .

The power of the low-rank model becomes evident in the absence of any part-of-speech tags .

For instance , on the English dataset , the low-rank model trained without POS tags achieves 90.49 % on first-order parsing , while the baseline gets 86.70 % if trained under the same conditions , and 90.58 % if trained with 12 core POS tags .

Finally , we demonstrate that the model can successfully leverage word vector representations , in contrast to the baselines .

Accurate scoring of syntactic structures such as head-modifier arcs in dependency parsing typically requires rich , high-dimensional feature representations .

We introduce a low-rank factorization method that enables to map high dimensional feature vectors into low dimensional representations .

Our method maintains the parameters as a low-rank tensor to obtain low dimensional representations of words in their syntactic roles , and to leverage modularity in the tensor for easy training with online algorithms .

We implement the approach on first-order to third-order dependency parsing .

Our parser outperforms the Turbo and MST parsers across 14 languages .

Future work involves extending the tensor component to capture higher-order structures .

In particular , we would consider second-order structures such as grandparent-head-modifier by increasing the dimensionality of the tensor .

This tensor will accordingly be a four or five-way array .

The online update algorithm remains applicable since each dimension is optimized in an alternating fashion .


Although multiword expressions frequently appear in daily communications, statistical machine translation models often produce wrong translations calling for special treatments of such expressions.

They propose a linguistically informed method for integrating phrasal verbs by adding features such as the presence of phrasal nouns, transitivity, separability and idiomaticity.

The proposed method outperforms similar models on an English to Bulgarian translation task evaluated by automatic and manual metrics showing the utility of introduced features.

Abstract: This article describes a linguistically informed method for integrating phrasal verbs into statistical machine translation ( SMT ) systems .

Abstract: In a case study involving English to Bulgarian SMT , we show that our method does not only improve translation quality but also outperforms similar methods previously applied to the same task .

Abstract: We attribute this to the fact that , in contrast to previous work on the subject , we employ detailed linguistic information .

Abstract: We found out that features which describe phrasal verbs as idiomatic or compositional contribute most to the better translation quality achieved by our method .

Phrasal verbs are a type of multiword expressions ( MWEs ) and as such , their meaning is not derivable , or is only partially derivable , from the semantics of their lexemes .

This , together with the high frequency of MWEs in every day communication ( see Jackendoff ( 1997 ) ) , calls for a special treatment of such expressions in natural language processing ( NLP ) applications .

Here , we concentrate on statistical machine translation ( SMT ) where the word-to-word translation of MWEs often results in wrong translations ( Piao et al . , 2005 ) .

Previous work has shown that the application of dedicated methods to identify MWEs and then integrate them in some way into the SMT process often improves translation quality .

Generally , automatically extracted lexicons of MWEs are employed in the identification step .

Further , various integration strategies have been proposed .

The so called static strategy suggests training the SMT system on corpora in which each MWE is treated as a single unit , e.g. call off .

This improves SMT indirectly by improving the alignment between source and target sentences in the training data .

Various versions of this strategy are applied in Lambert and Banchs ( 2005 ) , Carpuat and Diab ( 2010 ) , and Simova and Kordoni ( 2013 ) .

In all cases there is some improvement in translation quality , caused mainly by the better treatment of separable PVs , such as in turn the light on .

Another strategy , which is referred to as dynamic , is to modify directly the SMT system .

Ren et al . ( 2009 ) , for example , treat bilingual MWEs pairs as parallel sentences which are then added to training data and subsequently aligned with GIZA++ ( Och and Ney , 2003 ) .

Other approaches perform feature mining and modify directly the automatically extracted translation table .

Ren et al . ( 2009 ) and Simova and Kordoni ( 2013 ) employ Moses1 to build and train phrase-based SMT systems and then , in addition to the standard phrasal translational probabilities , they add a binary feature which indicates whether an MWE is present in a given source phrase or not .

Carpuat and Diab ( 2010 ) employ the same approach but the additional feature indicates the number of MWEs in each phrase .

All studies report improvements over a baseline system with no MWE knowledge but these improvements are comparable to those achieved by static methods .

In this article , we further improve the dynamic strategy by adding features which , unlike all previous work , also encode some of the linguistic properties of MWEs .

Since it is their peculiar linguistic nature that makes those expressions problematic for SMT , it is our thesis that providing more linguistic information to the translation process will improve it .

In particular , we concentrate on a specific type of MWEs , namely phrasal verbs ( PVs ) .

We add 4 binary features to the translation table which indicate not only the presence of a PV but also its transitivity , separability , and idiomaticity .

We found that PVs are very suitable for this study since we can easily extract the necessary informa-tion from various language resources .

To prove our claim , we perform a case study with an English to Bulgarian SMT system .

Bulgarian lacks PVs in the same form they appear in English .

It is often the case that an English PV is translated to a single Bulgarian verb .

Such manyto-one mappings cause the so called translation asymmetries which make the translation of PVs very problematic .

We perform automated and manual evaluations with a number of feature combinations which show that the addition of all 4 features proposed above improves translation quality significantly .

Moreover , our method outperforms static and dynamic methods previously applied to the same test data .

A notable increase in performance is observed for separable PVs where the verb and the particle(s ) were not adjacent in the input English sentence as well as for idiomatic PVs .

This clearly demonstrates the importance of linguistic information for the proper treatment of PVs in SMT .

We would like to point out that we view the work presented here as a preliminary study towards a more general linguistically informed method for handling similar types of translation asymmetries .

The experiments with a single phenomenon , namely PVs , serve as a case study the purpose of which is to demonstrate the validity of our approach and the crucial role of properly integrated linguistic information into SMT .

Our work , however , can be immediately extended to other phenomena , such as collocations and noun compounds .

The remainder of the paper is organised as follows .

Section 2 describes the asymmetries caused by PVs in English to Bulgarian translation .

Section 3 provides details about the resources involved in the experiments .

Section 4 describes our method and the experimental setup .

Section 5 presents the results and discusses the improvements in translation quality achieved by the method .

In this article , we showed that the addition of linguistically informative features to a phrase-based SMT model improves the translation quality of a particular type of MWEs , namely phrasal verbs .

In a case study involving SMT from English to Bulgarian , we showed that adding features which encode not only the presence of a PV in a given phrase but also its transitiveness , separability , and idiomaticity led to better translation quality compared to previous work which employs both static and dynamic strategies .

In future research , we will extend our method to other language pairs which exhibit the same type of translation asymmetries when it comes to PVs .

Such language pairs include , among others , English-Spanish and English-Portuguese .

Further , we will apply our linguistically informed method to other phenomena which cause similar issues for SMT .

Immediate candidate phenomena include other types of MWEs , collocations , and noun compounds .

When it comes to MWEs , we will pay special attention to the compositionality aspect since it seems to have contributed most to the good performance achieve by our method in the study presented here .


When prompting language models as a knowledge base, existing methods use actual sentences as queries while models may be sensitive to the form of questions.

They propose to treat prompts as sequences of vectors instead of words to achieve more flexibility and optimize them by gradient descent or fine-tuning.

The proposed approach outperforms previous methods across multiple English models and factual and commonsense tasks showing implicit factual knowledge has been underestimated.

Abstract: Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks , using a fill-in-theblank paradigm ( Petroni et al . , 2019 ) or a few-shot extrapolation paradigm ( Brown et al . , 2020 ) .

Abstract: For example , language models retain factual knowledge from their training corpora that can be extracted by asking them to " fill in the blank " in a sentential prompt .

Abstract: However , where does this prompt come from ?

Abstract: We explore the idea of learning prompts by gradient descent-either fine-tuning prompts taken from previous work , or starting from random initialization .

Abstract: Our prompts consist of " soft words , " i.e. , continuous vectors that are not necessarily word type embeddings from the language model .

Abstract: Furthermore , for each task , we optimize a mixture of prompts , learning which prompts are most effective and how to ensemble them .

Abstract: Across multiple English LMs and tasks , our approach hugely outperforms previous methods , showing that the implicit factual knowledge in language models was previously underestimated .

Abstract: Moreover , this knowledge is cheap to elicit : random initialization is nearly as good as informed initialization .

Abstract: x performed until his death in y .

Pretrained language models , such as ELMo ( Peters et al . , 2018 ) , BERT ( Devlin et al . , 2019 ) , and BART ( Lewis et al . , 2020a ) , have proved to provide useful representations for other NLP tasks .

Recently , Petroni et al . ( 2019 ) and Jiang et al . ( 2020 ) demonstrated that language models ( LMs ) also contain factual and commonsense knowledge that can be elicited with a prompt .

For example , to query the date-of- , " where we have filled the first blank with " Mozart , " and ask a cloze language model to fill in the second blank .

The prompts used by Petroni et al . ( 2019 ) are manually created , while Jiang et al . ( 2020 ) use mining and paraphrasing based methods to automatically augment the prompt sets .

Finding out what young children know is difficult because they can be very sensitive to the form of the question ( Donaldson , 1978 ) .

Opinion polling is also sensitive to question design ( Broughton , 1995 ) .

We observe that when we are querying an LM rather than a human , we have the opportunity to tune prompts using gradient descent-the workhorse of modern NLP-so that they better elicit the desired type of knowledge .

A neural LM sees the prompt as a sequence of continuous word vectors ( Baroni et al . , 2014 ) .

We tune in this continuous space , relaxing the constraint that the vectors be the embeddings of actual English words .

Allowing " soft prompts " consisting of " soft words " is not only convenient for optimization , but is also more expressive .

Soft prompts can emphasize particular words ( by lengthening their vectors ) or particular dimensions of those words .

They can also adjust words that are misleading , ambiguous , or overly specific .

Consider the following prompt for the relation date-of-death :

This prompt may work for the male singer Cab Calloway , but if we want it to also work for the female painter Mary Cassatt , it might help to soften " performed " and " his " so that they do not insist on the wrong occupation and gender , and perhaps to soften " until " into a weaker connective ( as Cassatt was in fact too blind to paint in her final years ) .

Another way to bridge between these cases is to have one prompt using " performed " and another using " painted . "

In general , there may be many varied lexical patterns that signal a particular relation , and having more patterns will get better coverage ( Hearst , 1992 ; Riloff and Jones , 1999 ) .

We therefore propose to learn a mixture of soft prompts .

We test the idea on several cloze language models , training prompts to complete factual and com-mon sense relations from 3 datasets .

Comparing on held-out examples , our method dramatically outperforms previous work , even when initialized randomly .

So when regarded as approximate knowledge bases , language models know more than we realized .

We just had to find the right ways to ask .

Well-crafted natural language prompts are a powerful way to extract information from pretrained language models .

In the case of cloze prompts used to query BERT and BART models for single-word answers , we have demonstrated startlingly large and consistent improvements from rapidly learning prompts that work-even though the resulting " soft prompts " are no longer natural language .

Our code and data are available at https:// github.com / hiaoxui / soft-prompts .

How about few-shot prediction with pretrained generative LMs ?

Here , Lewis et al . ( 2020b ) show how to assemble a natural language prompt for input x from relevant input-output pairs ( x i , y i ) selected by a trained retrieval model .

Allowing fine-tuned soft string pairs is an intriguing future possibility for improving such methods without needing to fine-tune the entire language model .


Existing pun detection works take a pipeline approach that is composed of detection and location while joint learning approaches perform well on other NLP tasks.

They propose to approach pun detection and location jointly as sequence labelling using a new tagging scheme that exploits structure information.

The proposed approach is effective for homographic and heterographic puns and achieves state-of-the-art performance and most puns appear in the second half of sentences.

Abstract: A pun is a form of wordplay for an intended humorous or rhetorical effect , where a word suggests two or more meanings by exploiting polysemy ( homographic pun ) or phonological similarity to another word ( heterographic pun ) .

Abstract: This paper presents an approach that addresses pun detection and pun location jointly from a sequence labeling perspective .

Abstract: We employ a new tagging scheme such that the model is capable of performing such a joint task , where useful structural information can be properly captured .

Abstract: We show that our proposed model is effective in handling both homographic and heterographic puns .

Abstract: Empirical results on the benchmark datasets demonstrate that our approach can achieve new state-ofthe-art results .

There exists a class of language construction known as pun in natural language texts and utterances , where a certain word or other lexical items are used to exploit two or more separate meanings .

It has been shown that understanding of puns is an important research question with various real-world applications , such as human-computer interaction ( Morkes et al . , 1999 ; Hempelmann , 2008 ) and machine translation ( Schröter , 2005 ) .

Recently , many researchers show their interests in studying puns , like detecting pun sentences ( Vadehra , 2017 ) , locating puns in the text ( Cai et al . , 2018 ) , interpreting pun sentences ( Sevgili et al . , 2017 ) and generating sentences containing puns ( Ritchie , 2005 ; Hong and Ong , 2009 ; Yu et al . , 2018 ) .

A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy , homonymy , or phonological similarity to another sign , for an intended humorous or rhetorical effect .

Puns can be generally categorized into two groups , namely hetero-graphic puns ( where the pun and its latent target are phonologically similar ) and homographic puns ( where the two meanings of the pun reflect its two distinct senses ) ( Miller et al . , 2017 ) .

( 1 ) When the church bought gas for their annual barbecue , proceeds went from the sacred to the propane .

The first punning joke exploits the sound similarity between the word " propane " and the latent target " profane " , which can be categorized into the group of heterographic puns .

Another categorization of English puns is homographic pun , exemplified by the second instance leveraging distinct senses of the word " gut " .

Pun detection is the task of detecting whether there is a pun residing in the given text .

The goal of pun location is to find the exact word appearing in the text that implies more than one meanings .

Most previous work addresses such two tasks separately and develop separate systems ( Pramanick and Das , 2017 ; Sevgili et al . , 2017 ) .

Typically , a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not , where all instances ( with or without puns ) are taken into account during training .

For the task of pun location , a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text , where the training data involves only sentences that contain a pun .

Therefore , if one is interested in solving both problems at the same time , a pipeline approach that performs pun detection followed by pun location can be used .

Compared to the pipeline methods , joint learning has been shown effective ( Katiyar and Cardie , 2016 ; Peng et al . , 2018 ) since it is able to re-duce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks .

In this work , we demonstrate that the detection and location of puns can be jointly addressed by a single model .

The pun detection and location tasks can be combined as a sequence labeling problem , which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag .

Since each context contains a maximum of one pun ( Miller et al . , 2017 ) , we design a novel tagging scheme to capture this structural constraint .

Statistics on the corpora also show that a pun tends to appear in the second half of a context .

To capture such a structural property , we also incorporate word position knowledge into our structured prediction model .

Experiments on the benchmark datasets show that detection and location tasks can reinforce each other , leading to new state-of-the-art performance on these two tasks .

To the best of our knowledge , this is the first work that performs joint detection and location of English puns by using a sequence labeling approach.1 2 Approach

In this paper , we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective .

We observe that each text in our corpora contains a maximum of one pun .

Hence , we design a novel tagging scheme to incorporate such a constraint .

Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase .

We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task , but was not explored in previous works .

Furthermore , unlike many previous approaches , our approach , though simple , is generally applicable to both heterographic and homographic puns .

Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective .

Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns .

Research on puns for other languages such as Chinese is still under-explored , which could also be an interesting direction for our future studies .