  Language models trained on billions of tokens have recently led to
unprecedented results on many NLP tasks. This success raises the question of
whether, in principle, a system can ever "understand" raw text without access
to some form of grounding. We formally investigate the abilities of ungrounded
systems to acquire meaning. Our analysis focuses on the role of "assertions":
contexts within raw text that provide indirect clues about underlying
semantics. We study whether assertions enable a system to emulate
representations preserving semantic relations like equivalence. We find that
assertions enable semantic emulation if all expressions in the language are
referentially transparent. However, if the language uses non-transparent
patterns like variable binding, we show that emulation can become an
uncomputable problem. Finally, we discuss differences between our formal model
and natural language, exploring how our results generalize to a modal setting
and other semantic relations. Together, our results suggest that assertions in
code or language do not provide sufficient signal to fully emulate semantic
representations. We formalize ways in which ungrounded language models appear
to be fundamentally limited in their ability to "understand".
