  Pretrained language models (LMs) such as BERT, RoBERTa, and ELECTRA are
effective at improving the performances of a variety of downstream NLP tasks.
Recently, researchers have incorporated domain and task-specific knowledge in
these LMs' training objectives and further enhanced models' capability of
handling downstream tasks. However, none of these LMs are designed specifically
for event temporal reasoning. We propose DEER, a language model that is trained
to focus on event temporal relations and performs better under low-resource
settings than original LMs. More specifically, we create a large number of
training samples to simulate the machine reading comprehension and information
extraction tasks for event temporal understanding and leverage a
generator-discriminator structure to reinforce the LMs' capability of event
temporal reasoning. Our experimental results show that DEER can achieve SOTA
results and works particularly well in low-resource settings across 5 widely
used datasets.
