  A recent work has shown that transformers are able to "reason" with facts and
rules in a limited setting where the rules are natural language expressions of
conjunctions of conditions implying a conclusion. Since this suggests that
transformers may be used for reasoning with knowledge given in natural
language, we do a rigorous evaluation of this with respect to a common form of
knowledge and its corresponding reasoning -- the reasoning about effects of
actions. Reasoning about action and change has been a top focus in the
knowledge representation subfield of AI from the early days of AI and more
recently it has been a highlight aspect in common sense question answering. We
consider four action domains (Blocks World, Logistics, Dock-Worker-Robots and a
Generic Domain) in natural language and create QA datasets that involve
reasoning about the effects of actions in these domains. We investigate the
ability of transformers to (a) learn to reason in these domains and (b)
transfer that learning from the generic domains to the other domains.
