Paper:
Strong inductive biases allow children to learn in fast and adaptable ways. Children use the mutual exclusivity (ME) bias to help disambiguate how words map to referents, assuming that if an object has one label then it does not need another. In this paper, we investigate whether or not standard neural architectures have an ME bias, demonstrating that they lack this learning assumption. Moreover, we show that their inductive biases are poorly matched to lifelong learning formulations of classification and translation. We demonstrate that there is a compelling case for designing neural networks that reason by mutual exclusivity, which remains an open challenge.

Review:
*** Increased to weak accept after discussion of merits of ME bias was improved in the paper *** 
This paper investigates whether neural networks exhibit a ‘mutual exclusivity (ME) bias’, whereby novel inputs tend to be associated with previously unseen outputs, an inductive bias that is cited to be present in children when learning to associate new words and objects. Via three different sets of experiments, the authors conclude that, under standard training procedures, neural networks in fact display an anti-ME bias: when faced with a novel input, they tend to assign less probability mass to unobserved outputs than is justified by the incoming data. The authors go on to argue that explicitly designing neural networks to reason by mutual exclusivity could lead to faster and more flexible learning. While the experiments across several domains convincingly show the presence of an anti-ME bias in neural network training, I recommend that the paper be rejected because it falls short in demonstrating to what extent performance could be improved by correcting or reversing this bias. The authors pose an interesting hypothesis, but it would gain a lot of credibility if they could provide an empirical analysis of an algorithm that uses ME reasoning to improve learning in a realistic setting or if they could at least perform some quantitative analysis of the effects of ME bias on task performance for a toy example.
Comments / questions:
* The authors duly acknowledge that ME bias is not necessarily desirable in all circumstances, namely in tasks that feature many-to-one mappings, citing polysemy and synonymy in machine translation as examples. Without an empirical example in a realistic setting, it is hard to judge whether the benefits of introducing an ME bias for better classification of new inputs belonging to new classes can outweigh the negatives via a potential increase in misclassification of those belonging to old classes.
* For the synthetic dataset one-to-one mapping of one-hot vectors it is clear that an ME bias would incur an advantage for classification on a zero-shot basis, i.e. an increased an accuracy for the first time a new input is observed. Is ME bias considered to be useful here because (i) we care directly about improving this type of zero-shot classification, or is it (ii) because it's implicitly assumed that a better initial prediction will lead to faster learning on these examples?
    * If it is (i) we care about, then it would be useful to quantify the advantage gained either empirically or analytically. It’s not obvious to me how useful the ME bias would be when there is a large number of classes since the probability assigned to the true novel class with a perfect ME bias would be 1 / (# unseen classes). So at the beginning of training, while there are lots of new classes to benefit from the ME bias, the maximum advantage per example is small, and vice versa at the end of training - how do these conflicting effects balance each other out over the course of training and how do the benefits of ME bias scale with the number of classes? Presumably, whether the benefit scales well will depend on the loss function - in any case, this warrants some analysis/discussion especially given that there are no experiments to test it.
    * The authors say that “ME can be generalized from applying to 'novel versus familiar’ stimuli to instead handling ‘rare versus frequent’ stimuli” and they cite the fact that neural networks take longer to learn from rare stimuli, suggesting that ME could help for reason (ii) above. It’s not obvious to me that ME bias will provide a significant advantage in a one-shot or few-shot learning setting - i.e. how useful is having made a better initial guess (which is random amongst the unseen classes) after you gain some specific information about the target class? Perhaps the benefit is small relative to that of the one-shot or few-shot information gained - I think this needs to be quantified empirically.
* It would be interesting to see a more detailed analysis of the predictions of the image classifiers in section 4.2. When a new image is presented, how is the probability mass distributed across previously seen classes versus across unseen ones? From a strategic standpoint, in a scenario where it is difficult to determine whether a new image belongs to a new class or not, could it plausibly be more sensible for the network to make a strong prediction on its best guess out of previously seen classes (that it knows more about) rather than a scattered prediction on the unseen classes? Admittedly this is a difficult question to quantify, but my point is to question whether the anti-ME bias shown in Figure 6 is necessarily suboptimal, given the difficulty of classifying a new image as a new class or not.
Minor comments /questions not affecting review:
* Is the acronym ME pronounced like the word “me” or is it spelled out “M-E”? If the latter, then all cases of “a ME bias” should be corrected to “an ME bias”.
* Section 4.2 line 3: “sample the class [for the] from a power law distribution"