What if Othello-Playing Language Models Could See? (2025)

Abstract: Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures.

## introduction does a language model truly understand what cat refers to? of course, none of us fully know what a cat is in an absolute sense, but human language users know enough to use the word appropriately. we can identify cats in images, infer that the furry, mouse-loving pet someone just described is likely a cat, and use the term in context with ease. whether mono-modal language models can achieve this level of grounding remains an open question. this paper does not aim to engage with the discussion over whether symbol grounding is in principle impossible for mono-modal language models @xcite . instead, we focus on the hypothesis that the inclusion of multiple modalities can facilitate more efficient learning. the question is orthogonal, but entirely consistent with the idea that mono-modal * equal contribution. to test this hypothesis, i.e., to what extent multimodal language models are more sample-efficient, we turn to the task of learning to play othello with language models, a domain that offers a welldefined, symbolic environment with clear rules and a compact action space, making it an ideal testbed @xcite . prior work has used this setup to investigate emergent world representations, training models ranging from smallscale language models @xcite to large language models (llms) @xcite to predict the next move from prior moves, with performance evaluated by next legal move accuracy to assess rule learning. a probing classifier is trained to investigate the representations learned for intermediate game states (e.g., my move vs. your move) @xcite . evidence suggests that language models can learn to track the board state, which potentially forms a rudimentary world model, when trained on large amounts of sequential data. while @xcite showed that text-only models can develop emergent world representations in othello, our work extends this framework into the multi-modal regime, by introducing vis-othello, an othello model trained on sequences of move histories and their corresponding board images (see figure 1 ). for each sequence of moves, we generate a corresponding sequence of board state images, with each image depicting the board at a specific time step. we then apply masking strategies to selected move tokens and train the model to predict the missing steps, using both the move history and associated visual context. our main goal is to investigate whether access to visual state information enhances sample efficiency and accelerates learning. we break down the main research questions into several related aspects: ## experiments in this section, we evaluate whether incorporating visual information improves learning efficiency and grounding in the othello game setting. ## semantically irrelevant perturbation to evaluate model robustness and generalization, we test performance under semantically irrelevant perturbations-input transformations that alter sur-face form without changing game state. ## feature alignment we perform representation alignment across models trained on othello game sequences to assess whether models trained on different modalities (i.e., image and text) learn similar representations. through this, we investigate whether modalityspecific models encode analogous patterns that are fundamental to rule-following gameplay. ## conclusion we studied the task of learning to play othello and extended it to a multi-modal setting by introducing visothello. our experiments examined whether access to visual state information improves sample efficiency and accelerates learning, comparing vis-othello against text-only and vision-only baselines. to further assess the benefits of multi-modal grounding, we introduce a board rotation perturbation and conduct feature alignment analysis to evaluate whether the models learn more robust and aligned representations. our findings suggest that grounding language models with visual input leads to more efficient and stable learning. beyond othello, our framework provides a controlled testbed for the analysis of grounded representations and has the potential to extend to other model architectures, tasks, and modalities (see appendix a for further discussion).