Paper: Zero and Few-shot Semantic Parsing with Ambiguous Inputs (ICLR-2024)

Reviewer comment (Evaluation):
I was curious about the focus on capturing the training data biases rather than the human annotation judgements, for the few-shot setting.

Author response:
Thanks for this point, it suggests an interesting direction for future work. In developing the FIM/FDM scores, we did consider whether to compare against human judgements. We opted to instead compare the range of possible ratios for two reasons. Firstly, we found in our human judgements that ambiguity is highly variable between examples, meaning we would need large amounts of data to make an accuracy comparison. For example, in Fig. 6 (now Fig 5.) PP attachment ambiguity examples range from 0 to 1. Averaging this to say that annotators generally think they have P(LF0) = 0.5 would obscure this variation, so we’d need instance-level annotations for all our evaluation examples. Secondly, ambiguity may not be a fixed quantity: we found that annotators differ confidently in their interpretations. Additionally, as pointed out by reviewer PRrS, there might be cultural differences in interpretation. Thus a fixed sample of interpretations might not be valid across different groups of people. We opted for metrics that subsume a metric based on estimates of judgments. Because we range the percentage of examples in the prompt from 0-100% in FDM and FIM, the true interpretation ratio should be found somewhere on this range. For example, if 30% of people interpret an example as LF0 and 70% interpret it as LF1, then this should be captured by FDM/FIM, since they test for this ratio. Marginalizing across ratios allows us to be agnostic about what the “true” ratio might be. Future work could expand this to a finer granularity – we considered increments of 10%, but given more examples we could consider finer increments.

Reviewer comment (Experiments):
The generation task is hard, especially generating logical forms. Why not formulate this as a multi-choice problem? Letting the model choose two from 10 possible combinations?

Author response:
This is a good question, which we have sought to address further based on your feedback. The reason to explore this problem semantic parsing (as opposed to multiple choice or open-ended language generation) is because of semantic parsing’s real-world applications. Parsing is used directly in real-world systems that have real consequences (e.g. robots, digital assistants, etc.). Ideally, these systems would be able to operate in open-ended environments, i.e. environments where we cannot frame the problem as multiple choice QA (since we don’t always know the choices, or there are too many to enumerate). Thus, while we’re interested in model’s performance on ambiguous language generally, we focus specifically on semantic parsing because of its practical applications. We agree that the results in a multiple choice setting may be very different from what we have presented, but these results would have little bearing on semantic parsing. We have added further motivation for exploring semantic parsing specifically in the introduction and in the discussion of implications of our work in the discussion (highlighted in blue).

Reviewer comment (Evaluation):
The evaluation metric can be improved. I have several questions about this. Why not use the same zero-shot and few-shot metric since the output format is the same? Why not use language interpretation instead of LF generations? Language interpretation is much more intuitive for the model than LF generations.

Author response:
1. *“ Why not use the same zero-shot and few-shot metric since the output format is the same?”*
The expected/desired behavior in the zero and few-shot settings is different. In the zero-shot setting, we don't know what the "true" output distribution is, so the metric just looks at whether the model puts enough mass on both interpretations to generate them. In the few-shot setting, we're giving the model evidence of the expected distribution, so we have a reference point, and can quantify whether the model captures that distribution. Thus, we need different metrics for the two settings. 
    2. *"Why not use language interpretation instead of LF generations? Language interpretation is much more intuitive for the model than LF generations."*
This is certainly true. However, in addition to the reason mentioned above for multiple choice (we care about semantic parsing, as opposed to general language generation) evaluation is very hard for language interpretation, and generally requires human evaluation, while automated evaluation for semantic parsing is easier since the form is constrained to a specific language.

Reviewer comment (Presentation):
Why not report ZM 100 in Figure 4?

Author response:
*“Why not report ZM 100 in Figure 4?”*
Decoding is computationally expensive to run. Typical semantic parsing systems use a beam-size of 5-10 for efficiency, so we follow past work here.

Reviewer comment (Experiments):
In section 3.1, the zero-shot experiment, how do you get multiple predictions? Do you sample k times? Your prompt doesn’t encourage the model to predict two LFs, so the model only predicts one by demonstration. The zero-shot results could get better if you encourage two predictions in your prompt.

Author response:
*“…The zero-shot results could get better if you encourage two predictions in your prompt.”*
Thanks for raising this point. While it’s true that we could likely perform better on ambiguous examples given a prompt for ambiguity, this prompt might be different from the overall best prompt for unambiguous examples. In that case, we would need some way of choosing the prompt based on the ambiguity in the example. In practice, semantic parsing systems typically use a single prompt for all examples. So here we use the same prompt as other past work ([Shin et al. 2021](https://arxiv.org/abs/2104.08768), [Shin and Van Durme 2022](https://arxiv.org/abs/2112.08696), [Roy et al. 2023](https://openreview.net/forum?id=k4juAEW1tG) )

Reviewer comment (Experiments):
Do you plan to extend the ambiguity to more than two interpretations?

Author response:
*“It's more like a suggestion: Do you plan to extend the ambiguity to more than two interpretations?”* 
This is a great suggestion. We are in fact working on this for an ongoing project – the AMP code does easily allow for >2 interpretations, though we choose to focus on 2 for the sake of clarity in this work.

Reviewer comment (Writing):
Section 2.1 Metrics. The last sentence is confusing: “The higher, the better.“ If you are talking about k values, shouldn't it be the lower, the better? If it refers to the metric, then it should be explicit.

Author response:
*“Section 2.1 Metrics. The last sentence is confusing: “The higher, the better. “ If you are talking about k values, shouldn't it be the lower, the better? If it refers to the metric, then it should be explicit.” and “Equation 1 is very confusing. Do i and k mean the same thing here?”* 
We have updated the description to clarify these points.

Reviewer comment (Writing):
Equation 1 is very confusing. Do i and k mean the same thing here?

Author response:
*“Section 2.1 Metrics. The last sentence is confusing: “The higher, the better. “ If you are talking about k values, shouldn't it be the lower, the better? If it refers to the metric, then it should be explicit.” and “Equation 1 is very confusing. Do i and k mean the same thing here?”* 
We have updated the description to clarify these points.

Reviewer comment (Presentation):
Figure 4 is a Table, not a Figure.

Author response:
*Section 3, paragraph 2, “Here”⇒ “In this setting” and “Figure 4 is a Table, not a Figure.”*
Thanks – we have fixed these issues.

Reviewer comment (Presentation):
Figure 5 is a bit hard to read when it comes to different types of semantic ambiguity. You can choose to put a different text color for important tokens inside the blue circle to increase the readability.

Author response:
*“Figure 5 is a bit hard to read when it comes to different types of semantic ambiguity. You can choose to put a different text color for important tokens inside the blue circle to increase the readability.”* 
Thanks for this suggestion. We have updated the text color based on the background color in this figure.

Reviewer comment (Experiments):
In Section 4, paragraph 2, what does “1 sentence at a time” mean when the percentage of the LF0 sentences is from 0 to 100 in increments of 10?

Author response:
*“In Section 4, paragraph 2, what does “1 sentence at a time” mean when the percentage of the LF0 sentences is from 0 to 100 in increments of 10?”*
Thanks for pointing this out – we meant that we vary the number of LF0 sentences in the prompt from 0 to 10, in increments of 1. Since there are 10 sentences total, this is the same as varying the percentage from 0 to 100 in increments of 10. We have updated the section to make this clearer.

Reviewer comment (Experiments):
Human experiments: humans were given both interpretations and asked to assign confidences to them. This seems a bit different from what the models were asked to do in the zero-shot experiments, which is implicitly pick out the ambiguity on their own. I understand it'd be hard to elicit this kind of behavior from humans — ideally you would ask them a question whose answer hinges on the ambiguity and get some measure of the uncertainty in their answer, but it's unclear what that measure would be. The measured degree of uncertainty from humans in this setting may quite exceed the actual uncertainty when reading and interpreting language in practice, as humans may simply snap to one reading or the other. So it may not be fair to compare this result to model results — though of course, there is no clear analogy between what humans and models are doing anyway, so I'm not sure what the exact criteria for a fair comparison should be. It might seem like a safer bet to compare humans to models on a similar task, where the model is given both interpretations and is asked to give a confidence score, or repeat the one that it prefers — for example, I think it'd be a strict improvement on the current paper to do the human comparison to the FIM relative confidence results instead. But of course, the confidences reported through that process may not be ecologically valid with respect to the paper's broader goal of characterizing LM behavior on ambiguous semantic parsing examples.

Author response:
We are in agreement that the human results are imperfect, and have added a section on limitations of our human judgements to the appendix contrasting our study with standard methods in psycholinguistics and providing further justification for our choices. Regarding a comparison between human judgments to FIM: as-is, this judgment would not be a completely fair comparison, since human annotators were not shown any examples, while the model is shown 10 ICL examples with a particular ratio of LF1 to LF2.

Reviewer comment (Evaluation):
More broadly, the paper relies on a mostly implicit assumption that the _appropriate_ behavior for an LM is to represent multiple possible readings in its top-k outputs, or for the probabilities of each possible reading to reflect the proportions of related readings in their prompts / ICL data. It isn't obvious to me that this is what we want or that it is the right way of approaching the question of whether LMs can handle ambiguity in semantic parsing appropriately. When prompted for a single output, why _should_ the model distribute its probabilities in any particular way? There's nothing that says that we have to use the top-k candidates in order to represent a set of semantically distinct alternatives, and indeed it is suboptimal for this purpose, as the paper notes that the top-k results are usually variations of the same interpretation. I can see how this may make sense if we were to, for example, use the LM probabilities as a prior to guide some kind of search through the space of parses; we then want all plausible interpretations to be discoverable. But it's not clear that's the best strategy in practice if we're going to be using LMs anyway: why not just ask it, for example, to decode all possible interpretations in sequence? There's a big space of possible things to do here if we want models to handle ambiguity, and unfortunately the results of these experiments only bear on a small subset whose promise is unclear. Not that the experiments done are bad or invalid, but I think the paper needs to make a clear argument about the nature of the construct being tested and what these results are supposed to inform.

Author response:
Thank you for pointing out this assumption. We do state the desired behavior of the model at the beginning of paragraph 3, but have added clarification about what the actual “model” is. As you astutely put it, having a distribution over interpretations makes sense if we are to “use the LM probabilities as a prior to guide some kind of search through the space of parses”. Since actual generations are being produced by beam search – which exactly is a search through the space of parses, guided by the LM’s probabilities – I think that we are in fact in agreement on the importance of the probabilities. We have added a point to introduction, clarifying that what is considered the “parser” is not only the LLM but also the search procedure used to actually decode parses (in our case, beam search). In terms of why we use beam search rather than sequentially decoding parses: we are aiming here to remain fairly close to how semantic parsing models are used in practice, rather than optimizing for performance on ambiguous examples specifically. We have clarified this in our limitations section.

Reviewer comment (Evaluation):
In Section 3.1, the statement "we rarely observe any models predicting both LFs correctly; one exception to this is conjunction ambiguity" seems also true for `bound`.

Author response:
Thanks for pointing this out – we have amended the text accordingly.

Reviewer comment (Writing):
In Section 6, the sentence "Making this assumption can reduce semantic parsing to syntactic parsing" is very unclear to me.

Author response:
We have clarified the point about semantic parsing being reduced to syntactic parsing. If there is a 1-to-1 mapping between syntax and meaning, then semantic parsing can be handled by a syntactic parser (this is of course not the case for real data).

Reviewer comment (Theory):
Even after author response, I remain concerned about the choice to ask that the model represent ambiguity in its top-k probabilities, which I think is a problematic assumption. I think this undermines the significance of the paper as it takes an outdated and limiting perspective on how to represent ambiguity, though I still think it constitutes a useful contribution to the literature.

Author response:
Thank you for pointing out this assumption. We do state the desired behavior of the model at the beginning of paragraph 3, but have added clarification about what the actual “model” is. As you astutely put it, having a distribution over interpretations makes sense if we are to “use the LM probabilities as a prior to guide some kind of search through the space of parses”. Since actual generations are being produced by beam search – which exactly is a search through the space of parses, guided by the LM’s probabilities – I think that we are in fact in agreement on the importance of the probabilities. We have added a point to introduction, clarifying that what is considered the “parser” is not only the LLM but also the search procedure used to actually decode parses (in our case, beam search). In terms of why we use beam search rather than sequentially decoding parses: we are aiming here to remain fairly close to how semantic parsing models are used in practice, rather than optimizing for performance on ambiguous examples specifically. We have clarified this in our limitations section.

Reviewer comment (Evaluation):
While the paper provides a strong foundation, it could benefit from a more detailed exploration of how ambiguity affects real-world applications of semantic parsing.

Author response:
We have added more details to the introduction, motivating why we examine semantic parsing and how misunderstandings could arise from ambiguity.

Reviewer comment (Experiments):
The AMP dataset, while novel, might still be limited in scope and diversity, potentially affecting the robustness of the study’s conclusions.

Author response:
This point is well-taken – while our study is an initial exploration into the underexplored area of ambiguity in semantic parsing, it is far from complete in terms of the range of possible ambiguities (and languages, settings, cultures, etc.) that could be explored. We have clarified our limitations subsection in Section 6 to emphasize that AMP can be extended both structurally (covering scope) and lexically (covering diversity).

Reviewer comment (Theory):
It is unclear how the proposed methods deal with the dynamic nature of conversational context, which can significantly affect ambiguity resolution.

Author response:
Thank you for pointing this out – this is definitely a limitation of our approach, since conversational context often reduces ambiguity. We have highlighted where we state this assumption in the intro and have added it to the limitations. Removing context is realistic in the “digital assistant” domain, since most people would not accept a digital assistant that is constantly listening, i.e. by design, assistants have access to only parts of conversations. Note that some ambiguity is likely to persist even with context, as ambiguity commonly leads to breakdown in human conversation.

Reviewer comment (Experiments):
Is the AMP dataset extensible, and are there plans to include more complex or nuanced forms of ambiguities, such as cultural or idiomatic ones?

Author response:
AMP is easily extensible – additional templates and vocabulary items can be added, and the code offers instructions on how to do so (see updated footnote 2). The suggestion to add cultural or idiomatic ambiguities sounds extremely interesting – could you give an example of a cultural ambiguity? Would this be an utterance that could be interpreted one way by a listener from one culture, and another way by a listener of another culture?

Reviewer comment (Experiments):
Could you elaborate on the selection process for the five types of natural language ambiguities included in your study? Were there other types of ambiguities considered but excluded?

Author response:
We selected our ambiguity types by reviewing the relevant literature in linguistics/psycholinguistics and pulling out well-documented and researched ambiguities. We did exclude lexical ambiguities from our study. Our rationale for this was that LLM models are fairly adept at word-sense disambiguation tasks, which have long been studied in NLP. To make examples non-trivial, we would have to come up with very contrived examples. Additionally, we found that lexical ambiguities only occur in very specific contexts, making generating data for them challenging. For example, a standard lexical ambiguity is “bank (monetary institution)” vs “bank (of a river)”. Some sentences allow for both (e.g. “I went to the bank” is ambiguous) but we couldn’t drop in another ambiguous noun, e.g. “rock (geology)” and “rock (music)” wouldn’t work in “I went to the rock”. In other words, each lexical ambiguity would need its own specific example, which is already covered by existing word-sense disambiguation data.

Reviewer comment (Experiments):
The use of synthetic data might not fully capture the complexity of natural language ambiguities encountered in real-world scenarios. How well do the findings translate to naturally occurring datasets?

Author response:
Because of a lack of real-world data on ambiguity, we don’t have quantitative results here. Speculatively, because we're using LLMs and 0-shot learning, the sim-to-real gap between synthetic and real data might be smaller. Normally, the problem with using synthetic data is that models trained on synthetic data don't generalize to real data. In our case, we don't train any models, and instead test a model trained on real data on synthetic data. While the results might differ, we would expect that (given that the model was trained mostly on real data) if it generalizes to synthetic data, it should also be able to generalize to real data, which is actually closer to the model's data distribution.