Embedding methods which enforce a partial order or lattice structure over the concept space , such as Order Embeddings ( OE ) ( Vendrov et al . , 2016 ) , are a natural way to model transitive relational data ( e.g. entailment graphs ) . However , OE learns a deterministic knowledge base , limiting expressiveness of queries and the ability to use uncertainty for both prediction and learning ( e.g. learning from expectations ) . Probabilistic extensions of OE ( Lai and Hockenmaier , 2017 ) have provided the ability to somewhat calibrate these denotational probabilities while retaining the consistency and inductive bias of ordered models , but lack the ability to model the negative correlations found in real-world knowledge . In this work we show that a broad class of models that assign probability measures to OE can never capture negative correlation , which motivates our construction of a novel box lattice and accompanying probability measure to capture anticorrelation and even disjoint concepts , while still providing the benefits of probabilistic modeling , such as the ability to perform rich joint and conditional queries over arbitrary sets of concepts , and both learning from and predicting calibrated uncertainty . We show improvements over previous approaches in modeling the Flickr and WordNet entailment graphs , and investigate the power of the model . Structured embeddings based on regions , densities , and orderings have gained popularity in recent years for their inductive bias towards the essential asymmetries inherent in problems such as image captioning ( Vendrov et al . , 2016 ) , lexical and textual entailment ( Erk , 2009 ; Vilnis and McCallum , 2015 ; Lai and Hockenmaier , 2017 ; Athiwaratkun and Wilson , 2018 ) , and knowledge graph completion and reasoning ( He et al . , 2015 ; Nickel and Kiela , 2017 ; Li et al . , 2017 ) . Models that easily encode asymmetry , and related properties such as transitivity ( the two components of commonplace relations such as partially ordered sets and lattices ) , have great utility in these applications , leaving less to be learned from the data than arbitrary relational models . At their best , they resemble a hybrid between embedding models and structured prediction . As noted by Vendrov et al . ( 2016 ) and Li et al . ( 2017 ) , while the models learn sets of embeddings , these parameters obey rich structural constraints . The entire set can be thought of as one , sometimes provably consistent , structured prediction , such as an ontology in the form of a single directed acyclic graph . While the structured prediction analogy applies best to Order Embeddings ( OE ) , which embeds consistent partial orders , other region-and density-based representations have been proposed for the express purpose of inducing a bias towards asymmetric relationships . For example , the Gaussian Embedding ( GE ) model ( Vilnis and Mc-Callum , 2015 ) aims to represent the asymmetry and uncertainty in an object 's relations and attributes by means of uncertainty in the representation . However , while the space of representations is a manifold of probability distributions , the model is not truly probabilistic in that it does not model asymmetries and relations in terms of prob-abilities , but in terms of asymmetric comparison functions such as the originally proposed KL divergence and the recently proposed thresholded divergences ( Athiwaratkun and Wilson , 2018 ) . Probabilistic models are especially compelling for modeling ontologies , entailment graphs , and knowledge graphs . Their desirable properties include an ability to remain consistent in the presence of noisy data , suitability towards semisupervised training using the expectations and uncertain labels present in these large-scale applications , the naturality of representing the inherent uncertainty of knowledge they store , and the ability to answer complex queries involving more than 2 variables . Note that the final one requires a true joint probabilistic model with a tractable inference procedure , not something provided by e.g. matrix factorization . We take the dual approach to density-based embeddings and model uncertainty about relationships and attributes as explicitly probabilistic , while basing the probability on a latent space of geometric objects that obey natural structural biases for modeling transitive , asymmetric relations . The most similar work are the probabilistic order embeddings ( POE ) of Lai ( Lai and Hockenmaier , 2017 ) , which apply a probability measure to each order embedding 's forward cone ( the set of points greater than the embedding in each dimension ) , assigning a finite and normalized volume to the unbounded space . However , POE suffers severe limitations as a probabilistic model , including an inability to model negative correlations between concepts , which motivates the construction of our box lattice model . Our model represents objects , concepts , and events as high-dimensional products-of-intervals ( hyperrectangles or boxes ) , with an event 's unary probability coming from the box volume and joint probabilities coming from overlaps . This contrasts with POE 's approach of defining events as the forward cones of vectors , extending to infinity , integrated under a probability measure that assigns them finite volume . One desirable property of a structured representation for ordered data , originally noted in ( Vendrov et al . , 2016 ) is a " slackness " shared by OE , POE , and our model : when the model predicts an " edge " or lack thereof ( i.e. P ( a|b ) = 0 or 1 , or a zero constraint violation in the case of OE ) , being exposed to that fact again will not update the model . Moreover , there are large degrees of freedom in parameter space that exhibit this slackness , giving the model the ability to embed complex structure with 0 loss when compared to models based on symmetric inner products or distances between embeddings , e.g. bilinear GLMs ( Collins et al . , 2002 ) , Trans-E ( Bordes et al . , 2013 ) , and other embedding models which must always be pushing and pulling parameters towards and away from each other . Our experiments demonstrate the power of our approach to probabilistic ordering-biased relational modeling . First , we investigate an instructive 2-dimensional toy dataset that both demonstrates the way the model self organizes its box event space , and enables sensible answers to queries involving arbitrary numbers of variables , despite being trained on only pairwise data . We achieve a new state of the art in denotational probability modeling on the Flickr entailment dataset ( Lai and Hockenmaier , 2017 ) , and a matching state-of-the-art on WordNet hypernymy ( Vendrov et al . , 2016 ; Miller , 1995 ) with the concurrent work on thresholded Gaussian embedding of Athiwaratkun and Wilson ( 2018 ) , achieving our best results by training on additional co-occurrence expectations aggregated from leaf types . We find that the strong empirical performance of probabilistic ordering models , and our box lattice model in particular , and their endowment of new forms of training and querying , make them a promising avenue for future research in representing structured knowledge . We have only scratched the surface of possible applications . An exciting direction is the incorporation of multi-relational data for general knowledge representation and inference . Secondly , more complex representations , such as 2n-dimensional products of 2-dimensional convex polyhedra , would offer greater flexibility in tiling event space . Improved inference of the latent boxes , either through better optimization or through Bayesian approaches is another natural extension . Our greatest interest is in the application of this powerful new tool to the many areas where other structured embeddings have shown promise .

Challenge: While a probabilistic extension can complement Order Embeddings with more flexibility, it still lacks the ability to model the negative correlation between concepts.
Approach: They first show that the existing probabilistic approach cannot capture negative correlation, and further propose a box lattice coupled with a probability measure.
Outcome: The proposed probabilistic order-biased relational model achieves state-of-the-art on the Flickr entailment dataset and WordNet hypernymy matching dataset.