A G erman WSC dataset comparing coreference resolution by humans and machines (2025)

Abstract: We present a novel German Winograd-style dataset for direct comparison of human and model behavior in coreference resolution. Ten participants per item provided accuracy, confidence ratings, and response times. Unlike classic WSC tasks, humans select among three pronouns rather than between two potential antecedents, increasing task difficulty. While majority vote accuracy is high, individual responses reveal that not all items are trivial and that variability is obscured by aggregation. Pretrained language models evaluated without fine-tuning show clear performance gaps, yet their accuracy and confidence scores correlate notably with human data, mirroring certain patterns of human uncertainty and error. Dataset-specific limitations, including pragmatic reinterpretations and imbalanced pronoun distributions, highlight the importance of high-quality, balanced resources for advancing computational and cognitive models of coreference resolution.

## introduction coreference resolution is a central task in nlp (for a review see @xcite , with most work focusing on fine-tuning models for benchmark performance (e.g., @xcite . in contrast, we directly compare the behavior of humans and pretrained language models (ptlms) on a task requiring coreference resolution. prior work shows that ptlms encode coreference-relevant biasessuch as preference for form similarity, recency, and grammatical agreement -when probed via contextual embeddings @xcite , mirroring patterns found in human anaphora resolution (e.g., @xcite . yet for direct human-machine comparison, analyzing ptlm behavior during sentence processing offers more insight than diagnostic probing. following ettinger (2020), we therefore assess ptlms in a psycholinguistic setup. we investigate how humans and ptlms process german winograd schemas coreference problems designed to test commonsense reasoning and named after an example in @xcite . the winograd schema challenge (wsc) @xcite was proposed as a more demanding alternative to the turing test @xcite ). foot_0 wsc items involve ambiguous pronouns whose resolution requires commonsense reasoning, and they are generally regarded as easy for humans but difficult for machines. a classic winograd schema (ws) consists of a pair of sentences differing only in a single critical word, that flips the intended referent of the pronoun. the classic task was to identify the correct antecedent: jane gave joan candy because she wasn't hungry. who [was/wasn't] hungry? [( ) jane; ( ) joan] in parallel experiments, we compare how humans and machines differ in processing coreference. specifically, we investigate (1) whether the same items are perceived as difficult by both groups, (2) which group performs better overall, and (3) whether model-based confidence measures (e.g., softmax probabilities) align with human selfassessed confidence ratings or response times. to ensure comparability, both groups perform the same task on the same data. since we are interested in the linguistic knowledge encoded by pretrained models rather than in their capacity for fine-tuning, we deliberately refrain from additional training. instead, we construct a dedicated dataset that allows direct human-machine comparison on items reflecting tasks already encountered during pretraining. ## experiments the key idea of our approach is to directly compare human and machine behavior on coreference resolution using a cloze-style task in german based on wsc items (see fig. 1 for an example). the original wsc dataset (wsc273) foot_1 comprises 273 manually constructed wsc pairs like those in (1), where the task is to choose between two potential antecedents. the pairs are designed to meet three criteria: (a) the correct referent is unambiguous for humans; (b) resolution cannot rely solely on selectional restrictions; and (c) frequency-based heuristics are insufficient. due to their difficulty and significance for machine translation and anaphora resolution, several larger wsc-style datasets have since been created. among them, winogrande @xcite is the most prominent, containing around 44,000 sentence pairs developed and validated via crowdsourcing. these are presented in cloze format, with the ambiguous pronoun replaced by a blank to be filled in, and two candidate antecedents provided as answer options. reported human accuracy on these datasets typically exceeds 90% or even 95% @xcite . our approach is related to @xcite , who tested the robustness of humans and ptlms on perturbed wsc items in cloze format (e.g., voice or tense changes), comparing majority vote (humans) and softmax predictions (ptlms). while they focused on accuracy and stability, we go further by comparing confidence levels. to ensure comparability between the human and machine experiments we (1) avoid task priming by using fillers (humans) and no fine-tuning (machines), and (2) present structurally identical items to both groups. ## results and discussion in the behavioral experiment, we find a moderate inter-annotator agreement among humans (κ = 0.562), with only 28 items answered unanimously. this relatively low agreement is itself an important finding. first, it challenges the common assumption that wsc items are straightforward for humans and thus constitute a reliable benchmark for evaluating machines. second, it raises concerns about the widespread practice of defining the human "gold" response via majority vote from as few as three annotators per item (see kocijan et al., 2023, for a survey). the observed lack of high interannotator agreement suggests that majority votes based on larger samples may yield substantially different outcomes. notably, for 21 items all three pronouns were chosen by at least one participant. at the same time, high agreement (≥7 of 10 participants selecting the same pronoun) was reached for 82 items, showing that while some items elicited highly consistent responses, a substantial number provoked genuinely ambiguous interpretations. among models, gbert-large and xlm-roberta-large perform comparably (accuracy ≈ 0.56), both outperforming the smaller bert-base-german-cased (accuracy ≈ 0.53). between configurations, accuracy remains largely stable, with top10 showing the highest target confidence, closely followed by pron, while top1 exhibits a notable drop. xlm-roberta-large achieves slightly higher target confidence than gbert-large and is therefore used in subsequent analyses. overall, model accuracy is somewhat lower than previously reported results on english wsc data (∼60%, kocijan et al., 2023), likely due to the increased complexity of our task: models perform free-form generation over the full token vocabulary and humans choose among three options (rather than two in classic wsc). human performance is considerably higher than model performance, but also reveals striking variability. while majority vote accuracy is relatively high (0.87), individual accuracy is markedly lower (@xmath0.729). this challenges the assumption that wsc-style problems are trivial for humans and highlight the limitations of majority-based metrics, potentially masking individual uncertainty. a breakdown by dataset reveals a strong qualin our analysis of human behavioral correlations, items with lower mean accuracy elicited longer mean response times (@xmath1.194, @xmath2.001) and lower mean confidence ratings (@xmath3.256, @xmath4.001). for the most extreme deciles mean accuracy raises from 0.65 for the slowest 10% of responses to 0.71 for the fastest, and from 0.49 for the lowest-rated items to 0.79 for the highest-rated ones, reinforcing the validity of these behavioral metrics. at the participant level, response time and confidence are themselves negatively correlated (@xmath5.142, @xmath6.001), indicating that individuals tended to take longer when less certain. comparing human and model behavior, we first note that all models predict the human majority vote more accurately than the gold answer ( correlations between model confidence (measured as softmax scores for both the gold and given answer) and human behavioral measures are shown in we observe several notable human and model error patterns. first, humans frequently select es to refer to an entire situation rather than one of the intended antecedents. excluding items with wrong majority vote es, majority vote accuracy rises to 0.97 and individual accuracy to 0.80, indicating that many apparent 'errors' are due to pragmatic reinterpretation (see appendix b for examples). second, models show markedly less variability across paired items. while humans gave identical answers to both items of a pair in 10 cases, models did so over 35 times (e.g., 37 in pron). this suggests a tendency to being biased and ignore subtle contextual shifts that differentiate minimal pairs. third, both humans and models exhibit systematic biases in antecedent selection. 4 human responses show a slight preference for the first antecedent (515 vs. 424), while models exhibit a stronger bias toward the second antecedent (e.g., pron: 57 vs. 41), reflecting the recency bias observed in probing studies @xcite . model accuracy is higher when the correct referent is the first antecedent (e.g., pron: 0.59 vs. 0.54), while humans perform better when the correct answer is in the second position (0.74 vs. 0.81). this asymmetry is challenging to interpret, as pronoun types are not evenly distributed across positions and gender biases may influence performance. for instance, models perform best on es (pron: 0.68), followed by sie (0.57) and er (0.46), while humans show a minor difference between er (0.72) and sie (0.71), and a pronounced advantage for es (0.80). ## conclusion we presented a novel german winograd-style dataset and collected fine-grained human data, including accuracy, confidence ratings, and response times, with 10 participants per item. 5 this resource provides a rich empirical basis for studying referential resolution in german and evaluating model behavior. thereby our task setup is more challenging than previous wsc formulations: humans must choose among three pronouns, and models face open-ended generation over the entire vocabulary. despite this, our results show clear human-machine performance gaps, alongside intriguing similarities in uncertainty and error patterns. at the same time, our analysis reveals limitations in the dataset itself: some 'errors' reflect pragmatic reinterpretations rather than misunderstanding. moreover, pronoun distribution is uneven across antecedent positions, suggesting room for improvement in future dataset design. taken together, our findings reinforce the critical importance of high-quality, carefully constructed data