In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.

For comparison purpose, we build two baseline systems for each direction: one is use the traditional phrase-based statistical machine translation (SMT), the other one is the NMT system.
Question: what was the baseline?
traditional phrase-based statistical machine translation (SMT) NMT system

For automatically evaluating our methods, we propose to use widely used metric for image/video captioning. This is because the proposed CommonGen task can be regarded as also a caption task where the context are incomplete scenes with given concept-sets. Therefore, we choose BLEU-3/4 BIBREF28, ROUGE-2/L BIBREF29, CIDEr BIBREF30, and SPICE BIBREF31 as the main metrics. Apart from these classic metrics, we also include a novel embedding-based metric named BERTScore BIBREF32. To make the comparisons more clear, we show the delta of BERTScore results by subtracting the score of merely using input concept-sets as target sentences, named $ riangle $BERTS.
Question: What automatic metrics are used for this task?
BLEU-3/4 ROUGE-2/L CIDEr SPICE BERTScore

A number of methods utilize an extra type of information as the observed features for entities, by either merging, concatenating, or averaging the entity and its features to compute its embeddings, such as numerical values BIBREF26 (we use KBLN from this work to compare it with our approach using only numerical as extra attributes), images BIBREF27 , BIBREF28 (we use IKRL from the first work to compare it with our approach using only images as extra attributes), text BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 , and a combination of text and image BIBREF35 . Further, BIBREF7 address the multilingual relation extraction task to attain a universal schema by considering raw text with no annotation as extra feature and using matrix factorization to jointly embed KB and textual relations BIBREF36 . In addition to treating the extra information as features, graph embedding approaches BIBREF37 , BIBREF38 consider observed attributes while encoding to achieve more accurate embeddings.
Question: What other multimodal knowledge base embedding methods are there?
merging, concatenating, or averaging the entity and its features to compute its embeddings graph embedding approaches matrix factorization to jointly embed KB and textual relations