ParallelQA-18: Multilingual Parallel QA Predictions and Human Validation Dataset
Authors: da Silva, Rafael, Eicher, Jeff
Type: Dataset
Published: 2026-08-03
Keywords: cross-lingual comprehension, multilingual NLP, low-resource languages, large language models, question answering, human evaluation, machine reading comprehension, benchmark reconstruction, CLCG, ParallelQA-18

<p><strong>Package v3.4.1 (bases-only):</strong> Packaging fix that ships a single Dataset archive. Scientific content is unchanged from v3.4.0 (non-identifying human-validation pair decoding, hashed QC exclusions, and error-analysis sampling/overlap provenance). It contains no copyrighted WOL article bodies, direct participant identifiers, paper aggregates, or source-of-truth result file. Companion Builder remains <code>10.5281/zenodo.21774412</code>. See <code>FLORES_ATTRIBUTION.md</code> and <code>COPYRIGHT_AND_SOURCE_TERMS.md</code>.</p><p>ParallelQA-18 is a reconstruction-based multilingual question-answering research resource developed to support the study of the Cross-Lingual Comprehension Gap (CLCG), a construct designed to measure systematic losses in semantic comprehension when the same content is presented to a language model in different languages.</p><p>The release covers 18 languages and provides the frozen research artifacts used in the associated study, including LLM-generated questions and gold answers, indexes for source-authored human questions, model responses by language, automatic metric scores, complete automated-judge outputs, anonymized human ratings, experimental configurations, sampling definitions, manifests, schemas, and derived CLCG estimates.</p><p>The resource is distributed as a reconstruction package rather than as a redistribution of the original corpus. Copyrighted source articles and source-authored human questions are therefore not included. Instead, the release provides the corresponding source URLs, publication and paragraph identifiers, extraction metadata, content hashes, and companion software required to retrieve and reconstruct the corpus locally from the original public source.</p><p>This design preserves the reproducibility of the study while respecting the copyright status of the underlying material. It also enables researchers to verify the provenance and integrity of reconstructed documents through stable identifiers and cryptographic hashes.</p><p>This record contains the frozen dataset snapshot used for the associated paper: internal study freeze v0.1 and public release v3.4.1, covering 150 articles, 5 evaluated language models, and 18 languages. Human-validation ratings from the completed collection (200 annotation batches) are included in anonymized form.</p><p><strong>Companion Software (required):</strong> This record is not intended to be used alone. It must be used together with the companion software <em>ParallelQA-18 Builder: Reconstruction and Evaluation Toolkit</em> (DOI: <a href="https://doi.org/10.5281/zenodo.21774412">10.5281/zenodo.21774412</a>). Record page: <a href="https://zenodo.org/records/21774412">https://zenodo.org/records/21774412</a>. The Dataset provides the frozen research artifacts; the Software provides the reconstruction, validation, and evaluation toolkit. Related-identifier metadata also links the two records.</p>