In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
--------
Question: We conduct experiments on our self-collected CAIS to evaluate the generalizability in different language. We apply two baseline models for comparison, one is the popular BiLSTMs + CRF architecture BIBREF36 for sequence labeling task, and the other one is the more powerful sententce-state LSTM BIBREF21. The results listed in Table TABREF50 demonstrate the generalizability and effectiveness of our CM-Net when handling various domains and different languages.
Question: What were the baselines models?

Answer: BiLSTMs + CRF architecture BIBREF36 sententce-state LSTM BIBREF21

Question: The split methods (beyond random split) are the following:

Output length: Variation of the setup described by BIBREF2 where the train set consists of examples with output (sparql query or action sequence) length $\le \hspace{-2.5pt} N$, while the test set consists of examples with output length $> \hspace{-2.5pt} N$. For CFQ, we use $N = 7$ constraints. For scan, we use $N = 22$ actions.

Input length: Variation of the above setup, in which the train set consists of examples with input (question or command) length $\le N$, while test set consists of examples with input length $> N$. For CFQ, we use $N=19$ grammar leaves. For SCAN, we use $N=8$ tokens.

Output pattern: Variation of setup described by BIBREF8, in which the split is based on randomly assigning clusters of examples sharing the same output (query or action sequence) pattern. Query patterns are determined by anonymizing entities and properties; action sequence patterns collapse primitive actions and directions.

Input pattern: Variation of the previous setup in which the split is based on randomly assigning clusters of examples sharing the same input (question or command) pattern. Question patterns are determined by anonymizing entity and property names ; command patterns collapse verbs and the interchangeable pairs left/right, around/opposite, twice/thrice.
Question: What are other approaches into creating compositional generalization benchmarks?

Answer: random Output length Input length Output pattern Input pattern

Question: To train our best model we chose the best network from our experiments (6-layer bLSTM with 1024 hidden units), trained it with Adam optimizer and fine-tuned with SGD with momentum using exponential learning rate decay.
Question: Which architecture is their best model?

Answer:
6-layer bLSTM with 1024 hidden units