Modeling Cross-Cultural Pragmatic Inference with Codenames Duet (2023)

Abstract: Pragmatic reference enables efficient interpersonal communication. Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers and listeners. In practice, however, speakers’ sociocultural background shapes their pragmatic assumptions. For example, readers of this paper assume NLP refers to Natural Language Processing, and not “Neuro-linguistic Programming.” This work introduces the Cultural Codes dataset, which operationalizes sociocultural pragmatic inference in a simple word reference game. Cultural Codes is based on the multi-turn collaborative two-player game, Codenames Duet. Our dataset consists of 794 games with 7,703 turns, distributed across 153 unique players. Alongside gameplay, we collect information about players’ personalities, values, and demographics. Utilizing theories of communication and pragmatics, we predict each player’s actions via joint modeling of their sociocultural priors and the game context. Our experiments show that accounting for background characteristics significantly improves model performance for tasks related to both clue-giving and guessing, indicating that sociocultural priors play a vital role in gameplay decisions.

## the cultural codes dataset this study has been approved by the institutional review board (irb) at the authors' institution. the purpose of the cultural codes dataset is to understand how measurable social factors influence dyadic communication in english. by collecting relevant participant background information, we aim to understand how these factors affect linguistic reasoning in a collaborative reference game. ## tasks and modeling to investigate the role of sociocultural factors in pragmatic inference, we propose a set of tasks ( ## experiment setup baselines and dataset splits for generation baselines, we use two seq2seq models: t5 @xcite and bart @xcite . we optimize the associated language modeling objective across our tasks. additionally, we experiment with two retrieval baselines for all generation tasks: for each task, we split clue givers into 80-10-10 train/val/test, since all tasks depend on initial clue giver choices. importantly, a single clue giver's data is not distributed across splits, since clue givers may reuse clues/strategies. evaluation metrics we use a range of metrics to generation tasks. rationale generation tasks (target §4.1.3 & guess §4.2.2) output entire sentences; therefore, we report f-1 scores from rouge-(1, 2, l) @xcite , bleu @xcite , and bertscore @xcite . for tasks that generate a single or set of words where order does not matter, (guess selection §4.2.1; clue generation §4.1.2), we report only rouge-1 and averaged word vector (fasttext) cosine similarity. ## generation results & discussion including cultural priors improves modeling performance across all tasks. for generation problems, t5 generally outperforms bart, and our retrieval baselines lag behind more complex models. finally, we conduct a qualitative analysis of 20 random samples from each task. picking targets and guesses from our results ( looking at model outputs between the none and morality, we observe that models generate words like well/grace instead of death/poison and vice versa, depending on player background. generating a clue for targets moving to our clue generation models, we again find that including sociocultural priors improves model performance ( while there are themes related to morality in clue differences for a target word (@xmath1. lucifer; or @xmath2. good), we also find that generations are more specific given sociocultural priors. consider these generated @xmath3: each ✓ example generates a clue that relies on shared cultural background: specifically, knowing that cricket is a sport; that james bond is a popular character; and that the undertaker is a wrestler. more details can be found in appendix c, clue generation errors across sociocultural subtypes despite jointly modeling cross-cultural information, our performance is far from perfect. generating successful clues is a core element of codenames; however, our exact match accuracy on clue generation is only ≈ 26%. to understand errors, we sample 100 generated clues from the clue generation task, and identify errors and differences between (socioculturally) generated clues and the ground truth label. for 43 samples, we notice that sociocultural priors have no effect on clue generation; the output is identical to the no prior model for the given target word. in these instances, we suspect that our models fail to exploit common ground between a giver/guesser, yielding the same clue as without sociocultural priors. upon further analysis, we observe that these errors occur frequently (37 samples) when both the clue giver and guesser are white or from north america. because these demographics are already over-represented in our dataset, we suspect that the model simply ignores over-informative sociocultural priors. errors also occur because clues are over (20 instances, e.g. "guevera" instead of "overthrow") or underspecified (13 instances, e.g. "supernatural" instead of "monster") compared to the gold clue. in 21/33 of these instances, there is a demographic mismatch between the clue-giver and guesser: the clue-giver and guesser do not share race/country demographics. in contrast to having no effect, we suspect that models mispredict the common ground between guesser/giver. we also judge 18 generation errors to be of similar specificity to the target word-prefixes/suffixes of the gold label-or completely unrelated to the gold clue (6 instances). rationalizing targets and guesses beyond generating target words and guesses, we ask models to explain how a target or guess is related to a clue word (e.g. james bond is a movie character). again, we find that providing contextual priors improves performance ( like with clue generation, we find that improvements in guess rationale are from increased specificity (e.g. "actors are cast" → "actors are part of a cast"; "money is center" → "money is the center of everything"). while qualitative differences are clear for guess rationale, target rationale results are more subtle: improvements stem from minor variations in the type of framing ("a kind of" vs. "a type of") used by the annotator. additional generations can be found in appendix c, classifying pragmatic failure we find that classification performance across each architecture is maximized when using sociocultural priors during training ( a word on word vector baselines surprisingly, retrieving nearest words using a word vector approach (fasttext) performs poorly for both clue and guess generation (tables 2 & 3 ). we suspect that pretrained vectors fail to capture sociocultural inference in word association tasks. ## conclusion language is grounded in rich sociocultural context. to underscore this context, we propose a setting that captures the diversity of pragmatic inference across sociocultural backgrounds. with our codenames duet dataset (7k turns across 156 players), we operationalize cross-cultural pragmatic inference. across our experiments, we detail improvements in mirroring/evaluating inferences when using sociocultural priors. our work highlights how integrating these priors can align models toward more socially relevant behavior.