Paper: Constrained Decoding for Cross-lingual Label Projection (ICLR-2024)

Reviewer comment (Experiments):
[Minor]: the work could be stronger if this could also be extended to larger models (e.g., >1-5B) and the discussion of applicability on more architectures (enc-only, enc-dec and dec-only) could be more details.

Author response:
Thank you for the suggestion, we have conducted more experiments with different sizes of the MT model used for constrained decoding in Codec and update the results in the Appendix C.3 of the paper.

Reviewer comment (Novelty):
Constrained decoding is also a large topic in semantic parsing and the authors could acknowledge work such as https://arxiv.org/abs/2109.05093

Author response:
Thank you for your insightful comments. We have edited the paper to fix the typos and acknowledge the work you suggested.

Reviewer comment (Presentation):
Math format mistake in F2 caption, k -> $k$

Author response:
Thank you for your insightful comments. We have edited the paper to fix the typos and acknowledge the work you suggested.

Reviewer comment (Experiments):
The approach relies on an external MT system, whose performance may influence the effectiveness of the label projection. It would be nice if there can be an analysis on the influence of translation quality.

Author response:
Thank you for your suggestion, we have conducted additional analysis about this topic and updated the paper with the new analysis in Appendix C.3. The update about this analysis is as follows. There are two places where MT systems are used in Codec: one is for generating the translation template, and the other one is for decoding. 
* We analyzed the impact of using different sizes of NLLB (600M, 1.3B, and 3.3B) to generate the translation templates, while fixed on NLLB-600M as the MT system for constrained decoding. We evaluate the methods in the translate-test setting on the MasakhaNER2.0 dataset, over 18 languages. Overall, the average F1 scores of methods using NLLB-600M, NLLB-1.3B and NLLB-3.3B as the template-translator are 68.6, 70.6 and 70.4 respectively. We observe the performance improves the most when changing from NLLB-600M to 1.3B, where NLLB’s translation quality also improves the most [1].
* Given the same translation templates (i.e., using NLLB-3.3B as the template-generator), we explored using Codec with the three model sizes of NLLB as the constrained decoding module. The average F1 score of methods using NLLB-600M, NLLB-1.3B, and NLLB-3.3B are 70.4, 70.1, and 70.1, respectively. 

[1] NLLB Team. No Language Left Behind: Scaling Human-Centered Machine Translation. https://arxiv.org/abs/2207.04672

Reviewer comment (Evaluation):
It would be nice to discuss and measure the efficiency of different methods, especially considering the extra stages of the proposed method. This can be important for “translate-test“, and maybe also for “translate-train” if the cost difference is too much.

Author response:
EasyProject is the fastest in the “translate-train” setup, as it does not require any additional step as Codec or Awesome-align; however, EasyProject is not applicable in the “translate-test” setup. Both Awesome-align and Codec can operate under the “translate-test” setup, which will entirely skip the training (fine-tuning) step of NER/EAE models on each target language as required by “translate-train”. We also provided a run time analysis for Codec in Figure 3b in the paper.

Reviewer comment (Experiments):
I’m wondering whether it would still be effective to replace the searching algorithm with some simpler alternatives, such as greedy pruning (like in a QA-MRC model). Since the problem itself is inserting a pair of markers, the output space is much smaller than the translation.

Author response:
We have implemented a simpler search algorithm, we dub it as Constrained-Space Beam Search (CSBS). In particular, this algorithm behaves similarly to the vanilla beam search, but we constrain the search space of it to contain only valid hypotheses - following the translation template and having the correct number of markers.  Similar to Codec, CSBS also returns up to 5 best hypotheses (equal to the beam size when the beam size is smaller than 5), and a re-ranking step is performed to choose the best hypothesis. Below is the performance of CSBS with different beam sizes and Codec on the MasakhaNER2.0 test set. The methods are evaluated in the translate-test setting (see below). We have updated these results in the Appendix E of the paper. Overall, a higher beam size will increase the performance of CSBS, however, even with a beam size of 16, CSBS’s average F1 score still falls behind Codec with a big gap. 

|  | CSBS | CSBS | CSBS | CSBS | Codec |
|-------------|--------|--------|--------|---------|-------|
| **Languages**   | beam=2 | beam=4 | beam=8 | beam=16 |  |
| Bambara     | 32.6   | 35.9   | 38.5   | 42.4    | 55.6  |
| Ewe         | 54.8   | 61.0   | 66.8   | 70.6    | 79.1  |
| Fon         | 21.3   | 24.2   | 29.0   | 32.7    | 61.4  |
| Hausa       | 54.4   | 66.4   | 68.7   | 70.1    | 73.7  |
| Igbo        | 46.2   | 52.3   | 56.5   | 60.3    | 72.8  |
| Kinyarwanda | 61.5   | 70.7   | 72.7   | 74.7    | 78.0  |
| Luganda     | 62.0   | 74.2   | 77.8   | 80.3    | 82.3  |
| Luo         | 27.4   | 31.8   | 34.4   | 41.1    | 52.9  |
| Mossi       | 22.2   | 25.3   | 29.4   | 34.5    | 50.4  |
| Chichewa    | 51.9   | 59.0   | 62.5   | 67.8    | 76.8  |
| chiShona    | 68.0   | 74.0   | 75.3   | 76.2    | 78.4  |
| Kiswahili   | 57.8   | 73.5   | 75.8   | 77.3    | 81.5  |
| Setswana    | 69.2   | 74.7   | 76.7   | 78.1    | 80.3  |
| Akan/Twi    | 57.3   | 63.1   | 65.4   | 68.1    | 73.5  |
| Wolof       | 39.1   | 45.8   | 50.7   | 54.7    | 67.2  |
| isiXhosa    | 49.9   | 59.9   | 64.2   | 67.0    | 69.2  |
| Yoruba      | 38.1   | 45.1   | 48.4   | 51.7    | 58.0  |
| isiZulu     | 54.2   | 69.4   | 72.2   | 74.6    | 76.9  |
| **Average**         | 48.2   | 55.9   | 59.2   | 62.3    | **70.4**  |

Reviewer comment (Experiments):
For the event task, it seems that the event triggers are assumed already given? How about considering the full event structures? This might be straight-forward since only span-projection would be enough (it would also be very interesting to consider pairs of spans when projecting).

Author response:
In our original submission, following many prior works, we used gold event triggers for the Event Argument Extraction (EAE) task, in order to do a more controlled experiment where variance is reduced due to errors in the first step of the pipeline for end-to-end event detection.  

We added a new experiment, in which an Event Trigger Detection model is first used to detect the event triggers; then given the input sentence and the detected triggers, the Event Argument Extraction model extracts the event arguments. For the Event Trigger Detection model, we fine-tune mDeBERTaV3-base on the concatenation of English data, and projected event trigger data in Arabic/Chinese. Details about the experiments and results have been updated in Appendix D.1 of the paper. Below are the results of different methods in this pipeline setting:

| Languages | Tasks | FT_En | Awes-align | EasyProject | Codec |
|-----------|-------|-------|------------|-------------|-------|
| Arabic    | ETD    | 48.0  | **54.6**       | 47.2        | 51.3  |
|           | EAE   | 26.7  | 33.1       | 26.9        | **33.6**  |
|           | Average   | 37.4  | **43.9**       | 37.1        | 42.5  |
| Chinese   | ETD    | 52.8  | **54.4**       | 51.5        | 53.1  |
|           | EAE   | 33.5  | 37.1       | 37.2        | **41.2**  |
|           | Average  | 43.2  | 45.8       | 44.4        | **47.2**  |

("ETD": Event Trigger Detection, "EAE": Event Argument Extraction, "FT_En": Fine-tune the model on English data only)

Overall, Awesome-Align (an embedding-based word alignment method) is a very strong baseline for high-resource languages, such as Arabic and Chinese. Codec is a marker-based method, which is more directly comparable to EasyProject (the state-of-the-art marker-based method). Codec's strengths are more apparent for low-resource languages and event arguments (i.e., varied nouns, pronouns, noun phrases, and entity mentions) where the multilingual LLM/embeddings are less performant. Event triggers are often the main verbs or action nouns (e.g., “married”, “attack”), where word aligner seems to be doing better, especially in high-resource languages. The results are a little bit more mixed, as the errors can happen in either trigger detection step (then propagate) or the event argument step, or both, which cause more variances (i.e., a less controlled experiment than when gold triggers are given).

Reviewer comment (Novelty):
The author say that inserting markers would degrade the translation quality. However, although they provide a translation template to guide the model during translation, the proposed method still relies on markers, which is not completely solving this issue.

Author response:
There seems to be some confusion here, our proposed method, Codec, does not include markers during the translation phase; so, the final translated text quality from Codec will be the same as any MT system, which is also the same as the word alignment-based label projection approaches. Only after this translation phase (i.e., after we generated this translation template), the markers are inserted through a second constrained decoding phase, in which the translation is constrained to be the same as the original translation template, while markers are being inserted. We have updated the Introduction of the paper to make this point clearer.

Reviewer comment (Experiments):
When searching, they mention the assumption:
```
If we decode the translation template but conditioned on the marker sentence, at the position that needs to be inserted an opening marker, the model would give a high probability to this token, and thus assign a low probability to the token from the template.
```
This assumption largely relies on the translation model’s ability to handle markers. Different translators may have different behaviors to handle the markers. I suggest the authors to report the results of different translators to show the stability of the proposed method.

Author response:
Thank you for your suggestion. We have added experiments with two more different MT systems (M2M-100 and MBart), besides NLLB.

* For the NER task, we explore using M2M-100 instead of NLLB for decoding. M2M-100 is chosen for NER as it supports 6 African languages in the MasakhaNER2.0 dataset. In addition, for this task, we also experiment with different model sizes of NLLB (i.e., 600M, 1.3B, and 3.3B) as the decoding module for Codec. See the table below (under “Codec” column) for results under the translate-test setting.  We have also included the results of these analyses in Appendix C of the paper. 

|     | FT_En | CSBS | CSBS | CSBS | CSBS | Codec | Codec | Codec | Codec |
|--------------|-------|--------|--------|--------|---------|-----------|-----------|---------|--------|
| Languages    |  | beam=2 | beam=4 | beam=8 | beam=16 | NLLB-600M | NLLB-1.3B | NLLB-3B | M2M-100 |
| Bambara      | 37.1  | 32.6   | 35.9   | 38.5   | 42.4    | 55.6      | **56.0**      | 55.8    | -      |
| Ewe          | 75.3  | 54.8   | 61.0     | 66.8   | 70.6    | 79.1      | **79.3**      | 77.7    | -      |
| Fon          | 49.6  | 21.3   | 24.2   | 29.0   | 32.7    | 61.4      | **62.4**      | 60.2    | -      |
| Hausa        | 71.7  | 54.4   | 66.4   | 68.7   | 70.1    | **73.7**      | 73.5      | 73.1    | 73.1   |
| Igbo         | 59.3  | 46.2   | 52.3   | 56.5   | 60.3    | 72.8      | **72.9**      | 72.8    | 72.8   |
| Kinyarwanda  | 66.4  | 61.5   | 70.7   | 72.7   | 74.7    | **78.0**      | 71.3      | 71.6    | -      |
| Luganda      | 75.3  | 62.0     | 74.2   | 77.8   | 80.3    | 82.3      | **82.4**      | 81.7    | -      |
| Luo          | 35.8  | 27.4   | 31.8   | 34.4   | 41.1    | 52.9      | **53.1**      | **53.1**    | -      |
| Mossi        | 45.0  | 22.2   | 25.3   | 29.4   | 34.5    | 50.4      | **52.1**      | 50.7    | -      |
| Chichewa     | 79.5  | 51.9   | 59.0   | 62.5   | 67.8    | 76.8      | 76.7      | **76.9**    | -      |
| chiShona     | 35.2  | 68.0     | 74.0   | 75.3   | 76.2    | **78.4**      | 78.0      | 78.3    | -      |
| Kiswahili    | 87.7  | 57.8   | 73.5   | 75.8   | 77.3    | 81.5      | 82.0      | 82.5    | **82.9**   |
| Setswana     | 64.8  | 69.2   | 74.7   | 76.7   | 78.1    | 80.3      | 81.0      | **81.2**    | -      |
| Akan/Twi     | 50.1  | 57.3   | 63.1   | 65.4   | 68.1    | 73.5      | 72.7      | **74.6**    | -      |
| Wolof        | 44.2  | 39.1   | 45.8   | 50.7   | 54.7    | 67.2      | 66.2      | **67.5**    | -      |
| isiXhosa     | 24.0    | 49.9   | 59.9   | 64.2   | 67.0    | 69.2      | **69.6**      | 69.1    | 67.9   |
| Yoruba       | 36.0    | 38.1   | 45.1   | 48.4   | 51.7    | 58.0      | 56.5      | **58.7**    | 53.8   |
| isiZulu      | 43.9  | 54.2   | 69.4   | 72.2   | 74.6    | **76.9**      | 76.6      | 76.6    | 75.7   |
| **Average**          | 54.5  | 48.2   | 55.9   | 59.2   | 62.3    | **70.4**      | 70.1      | 70.1    |    -    |

(‘-’: these low-resource languages are not supported by the M2M-100 MT system)

* For the EAE task, we have conducted additional experiments using MBart50-large (a competitive MT system for Chinese and Arabic) for both translation template generation and constrained decoding in Codec, as well as other comparison baselines. See table below for results. We have updated the paper (Appendix D.2) to include these results.

 Languages   | FT_En | Awes-align | EasyProject | Codec |
|---|---|---|---|---|
| Arabic  | 44.8  | 49.1  | 45.4  | **49.5** |
| Chinese | 54.0  | 56.6 | **57.6** | 57.2 |
| **Average**     | 49.4  | 52.9 | 51.5 | **53.4** |

Reviewer comment (Theory):
The proposed method is based on some heuristic. It would be great if the authors can provide some theoretical bound to justify the heuristic.

Author response:
Similar to many papers on NLP research, we acknowledge that we focus on empirical research, design and experiments for new methods. We agree that theoretical analysis of our new approach to constrained decoding based on branch-and-bound search would be valuable, however we view this as beyond the scope of this paper. That said, inspired by your comment, we have added an experiment to compare Codec to a variant of Beam Search, we dub it as Constrained-Space Beam Search (CSBS). This algorithm behaves similarly to the vanilla beam search, but we constrain the search space of it to contain only valid hypotheses -- following the translation template and having the correct number of markers. We have reported the performance of CSBS with different beam sizes and Codec in the translate-test setting of the MasakhaNER2.0 (please refer to the table in our reply to your second concern). In summary, higher beam size will increase the performance of CSBS, however, even with the beam size of 16, CSBS’s average F1 score still falls behind Codec with a big gap. We have updated these results in the Appendix E of the paper.

Reviewer comment (Experiments):
I am a little bit confused by the reason for not considering translation-test for EAE. The authors mention that the English gold trigger is needed but not feasible. However, this can be obtained by applying the proposed constraint label project from the target language to English. It would be interesting to study more about this.

Author response:
We agree that the English trigger can be achieved by using another label projection from the target language to English. We have updated the Experiment section in the paper to make this point clearer.

Reviewer comment (Experiments):
The main issue with the work is the lack of recognition of other (recent and less recent) work on the same problem of cross-lingual label projection, which consequently leads to the lack of more comprehensive comparisons to more baselines. The main baseline is definitely the EasyProject method and I agree with that, but I feel that not enough care has been provided to optimise the word alignment-based baselines which also shows reasonable performance, and is quite competitive in the EAE task.

Author response:
For 1), thank you for pointing this out. There is indeed a long history of research on cross-lingual label projection. We should cite as many papers as possible as we can fit in the page limit. We have updated our Related Work section to include all the papers you mentioned. 

For 2), thank you for your suggestion. Our paper focuses on improving the marker-based approach for label projection (especially low-resource languages), thus EasyProject, the best marker-based approach is used as the most important baseline for comparison as you mentioned. In terms of comparison to the word alignment-based approaches, in our original submission, we followed the prior work [1] in choosing Awesome-align (one of the state-of-the-art word alignment models) with multilingual BERT as the baseline. In response to your comments, we have added two more word-alignment baselines that are optimized for African languages, namely Awesome-align with AfroXLMR-large [2], a model that is adapted to 17 African languages, and Awesome-align with Glot500 [3], a model which is pre-trained on 500+ languages including African languages. These are novel and very competitive word alignment-based baselines for the MasakhaNER2.0 dataset. 

The table below compares the fine-tuning of English data baseline (FT_En), and Awesome-align with multilingual BERT (result in the main paper), Awesome-align with two aforementioned pre-trained models, and our proposed Codec in the translate-test setting, the results are reported on the test set of MasakhaNER2.0 dataset. To save space, in this table, we also include the experiments of using Codec with different MT models (NLLB and M2M-100) for decoding, which is to respond to your later comments. We have updated the paper (Appendix C.1) to include these results.

| | FT_En | Awes-align | Awes-align | Awes-align | Codec | Codec|
--- | --- | --- | --- | --- | ---|---|
**Languages** || mBERT | AfroXLMR | Glot500 | NLLB | M2M-100
Bambara | 37.1 | 50.0 | 53.8 | 52.0 | **55.6** | -
Ewe | 75.3 | 72.5 | 78.0 | 76.8 | **79.1** | -
Fon | 49.6 | 62.8 | **66.0** | 61.2 | 61.4 | -
Hausa | 71.7 | 70.0 | **73.7** | 73.1 | **73.7** | 73.1
Igbo | 59.3 | **77.2** | 71.7 | 71.2 | 72.8 | 72.8
Kinyarwanda | 66.4 | 64.9 | 68.2 | 68.1 | **78.0** | -
Luganda | 75.3 | 82.4 | 82.1 | **82.5** | 82.3 | -
Luo | 35.8 | 52.6 | 51.3 | 51.2 | **52.9** | -
Mossi | 45.0 | 48.4 | 49.1 | 48.7 | **50.4** | -
Chichewa | **79.5** | 78.0 | 76.7 | 77.3 | 76.8 | -
chiShona | 35.2 | 67.0 | **78.6** | 75.6 | 78.4 | -
Kiswahili | **87.7** | 80.2 | 80.5 | 79.5 | 81.5 | 82.9
Setswana | 64.8 | **81.4** | 80.7 | 80.5 | 80.3 | -
Akan/Twi | 50.1 | 72.6 | 71.7 | **73.5** | **73.5** | -
Wolof | 44.2 | 58.1 | 59.0 | 57.1 | **67.2** | -
isiXhosa | 24.0 | 52.7 | 67.6 | 63.9 | **69.2** | 67.9
Yoruba | 36.0 | 49.1 | 52.7 | 49.2 | **58.0** | 53.8
isiZulu | 43.9 | 64.1 | 75.5 | 74.9 | **76.9** | 75.7
**Average** | 54.5 | 65.8 | 68.7 | 67.6 | **70.4**| -

(‘-’ marks the low-resource languages that are not supported by the M2M MT system)

[1] Chen et al. Frustratingly easy label projection for cross-lingual transfer. Findings ACL 2023.

[2] Alabi et al. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. COLING 2022.

[3] ImaniGooghari et al. “Glot500: Scaling multilingual corpora and language models to 500 languages”. ACL 2023.

Reviewer comment (Novelty):
There has been some recent work on alignment correction for label projection (https://aclanthology.org/2023.law-1.24.pdf), and there are also other very relevant papers which should be cited and discussed (and ideally even compared against): https://aclanthology.org/2021.findings-acl.396.pdf or https://d-nb.info/1203127499/34.

Author response:
For 1), thank you for pointing this out. There is indeed a long history of research on cross-lingual label projection. We should cite as many papers as possible as we can fit in the page limit. We have updated our Related Work section to include all the papers you mentioned.

Reviewer comment (Experiments):
The number of evaluation tasks is slightly underwhelming and the paper should extend the scope of tasks to other sequence labeling tasks (e.g., slot labeling in dialogue, dependency parsing or semantic role labeling). NER with only 3 NE classes is a (relatively) simple task (from the perspective of its experimental setup), and the paper would have more impact with a wider experimental setup.

Author response:
Our proposed work primarily targets low-resource languages and span-level NLP tasks, where human-annotated datasets (that are directly annotated on these languages, other than annotating the translated texts from existing English datasets) are in scarcity. MasakhaNER2.0, which covers 18 African languages we can experiment on, is one of the best benchmarks that is available. Besides the NER task, we also did experiments on the ACE 2005 dataset for the EAE task, which is a commonly used benchmark in the label projection literature. We politely argue that having conducted experiments in 20 languages and two tasks (with 3 different MT systems, and 4 competitive baselines -- see our updates) are sufficient to support our claimed contributions, while agreeing with you that it is always nice to include even more experiments if possible.

Reviewer comment (Experiments):
I would like to see a wider exploration of different MT systems and chosen encoder-decoder models and their impact on the performance of both alignment-based approaches as well as EasyProject and CODEC. For instance, how would larger variants of NLLB affect the performance? Would the scale of the NMT system recover for its deficiencies?

Author response:
- We have added experiments with two more open-sourced MT systems. For the cross-lingual NER task, we use M2M-100 in place of NLLB as the MT model for decoding, and report the performance of this approach in the translate-test setting, on the MasakhaNER2.0 dataset. Please refer to the table at the beginning of our response for the added results. In summary, Codec using M2M-100 as the decoding module has similar performance Codec using NLLB on most of the languages, except for isiXhosa, and also outperforms the baseline of fine-tuning on English data (FT_En) on 5 over 6 supported languages. We have updated these results in the Appendix C.2 of the paper.

    For the Event Argument Extraction task, we have also conducted additional experiments of using MBart50-large (a competitive MT system for Chinese and Arabic) for both translation template generation and constrained decoding in Codec, as well as comparison baselines. See table below. We have updated the paper (Appendix D.2) to include these results. 

| Lang.   | FT_En | Awes-align | EasyProject | Codec |
|---------|-------|----------------------|------------|-------------|
| Arabic  | 44.8  | 49.1                 | 45.4       | **49.5** |
| Chinese | 54.0  | 56.6                 | **57.6**       | 57.2 |
| Average     | 49.4  | 52.9                 | 51.5       | **53.4** |

- In terms of the scales of NMT systems, we want to clarify that, in our original submission, both the Awesome-align and EasyProject baselines were using one of the larger NMT models (NLLB-3.3B). In response to your comment, we further added some experiments (each number is averaged from 18 languages) with different scales of NMT models for Codec on MasakhaNER2.0 in the Translation-test setup:

    - Different scales of MT for template generation in Codec (when the decoder is fixed to NLLB-600M): the average F1 scores of NLLB-600M, NLLB-1.3B, and NLLB-3.3B are 68.6, 70.6, and 70.4, respectively. 

    - Different scales of MT for decoding in Codec (when the template is fixed to that generated by NLLB-3.3B): the average F1 scores of NLLB-600M, NLLB-1.3B, and NLLB-3.3B are 70.4, 70.1, 70.1, respectively. 

    We have included the above results in Appendix C.3 of the paper.