You will be given a definition of a task first, then some input of the task.
In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.

In our final system, after pre-training the forward and backward LMs separately, we remove the top layer softmax and concatenate the forward and backward LM embeddings to form bidirectional LM embeddings, i.e., INLINEFORM0 .
Question: how are the bidirectional lms obtained?
Output:
They pre-train forward and backward LMs separately, remove top layer softmax, and concatenate to obtain the bidirectional LMs.