F ic S im: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction (2025)

Abstract: As language models become capable of processing increasingly long and complex texts, there has been growing interest in their application within computational literary studies. However, evaluating the usefulness of these models for such tasks remains challenging due to the cost of fine-grained annotation for long-form texts and the data contamination concerns inherent in using public-domain literature. Current embedding similarity datasets are not suitable for evaluating literary-domain tasks because of a focus on coarse-grained similarity and primarily on very short text. We assemble and release a dataset, FicSim, of long-form, recently written fiction, including scores along 12 axes of similarity informed by author-produced metadata and validated by digital humanities scholars. We evaluate a suite of embedding models on this task, demonstrating a tendency across models to focus on surface-level features over semantic categories that would be useful for computational literary studies tasks. Throughout our data-collection process, we prioritize author agency and rely on continual, informed author consent.

## introduction the last few years have been a time of immense progress in long-context processing in nlp. several language models now support context lengths in excess of a million tokens. embedding models for 32k context inputs abound. while challenges remain in long context modeling, successive approaches have made strong progress on benchmarks and have found applications downstream @xcite @xcite . 1 dataset can be accessed at https://huggingface.co/datasets/ficsim/ficsim . additional documentation can be found at https://github.com/natashamariejohnson330/ficsim figure 1 : similarity between literary texts can be defined along many dimensions. computational literary studies scholars generally seek to measure specific, semantic types of similarity such as similarity in tone, but embedding models over-index on more obvious features such as the author's writing style. in parallel, there is interest in applying nlp methods within digital humanities (dh), particularly within computational literary studies. many dh scholars have incorporated nlp methods such as topic modeling, sentiment analysis, and semantic textual similarity (sts) tasks such as clustering and measuring similarity into their research @xcite . yet evaluations of these approaches have been limited, particularly with regards to sts tasks. many nlp models which have been (or have the potential to be) applied in dh research are not evaluated on literary applications. when they are, it is often on digital texts made accessible through public repositories such as project gutenberg @xcite et al., 2019; @xcite @xcite . however, these texts-alongside related data and analyses from other commonly-scraped sites like wikipedia-are included in the pretraining data of most models @xcite , which could cause direct or indirect contamination and skew evaluation results @xcite . furthermore, while existing literary datasets evaluate model suitability for tasks such as summarization @xcite , question answering @xcite , and identifying literary co-reference @xcite , dh scholars have expressed the need for embedding methods that capture semantic textual similarity within novel-length texts along several axes such as plot, tone, and setting @xcite . in response to this gap, we present ficsim, an evaluation dataset for fine-grained semantic textual similarity (sts), constructed of long-form humanwritten narratives that are unlikely to appear in pretraining data, are accompanied by author-labeled metadata, and are included in this dataset with author consent. we describe our processes for selecting text not included in commoncrawl scrapes, for obtaining and maintaining author consent for the use of their works, and for constructing pairwise similarity measurements corresponding to 12 different facets of fictional texts, in consultation with both literary scholars and authors ( §3). we then describe the resulting dataset and its use ( §3.3). we evaluate existing approaches on this multi-faceted sts task ( §4). models struggle to capture salient characteristics of long-form texts, only weakly disambiguate between categories, and over-index on surface features of the text ( §5). we conclude by discussing the relevance of our results for both literary studies scholars and nlp researchers ( §7). we hope that ficsim enables more focus on narrowing the gap between models' general capabilities and their applicability to literary domain tasks. ## evaluation for each model, we compute cosine similarity between embeddings for all story pairs. following @xcite , we measure spearman's ρ between the model-induced ranking of story similarities and our tagset-derived gold ranking. we report ρ out of 100 instead of 1.0 for readability, and highlight values that are significant (@xmath0.05). models. we select the 3 best-performing openweights models on the hugging face @xcite mteb leaderboard with 32k context lengths: linq-embed-mistral @xcite , gte-qwen2-7b-instruct @xcite , and sfr-embed-mistral @xcite . these are all 7b models; however, computational literary studies scholars often have access to limited computational resources. thus, we consider a much smaller model-gte-modernbert-base @xcite )-and two api-based solutions: m2-bert-80m-32k-retrieval @xcite through the togetherai api and using voyage-3-large (voyage ai, 2025) through the voyage.ai api. finally, we consider whether large language models could perform this task. we use claude-3.7-sonnet (anthropic, 2025) to summarize each story, then embed the much shorter summary documents with voyage. 9 we follow each model's default strategy for constructing embeddings; for additional details on models and pooling methods, see appendix b. prompt. linq-embed, sfr-embedding, and both gte models support providing an instruction in a special format at the beginning of a text to be embedded. we experiment with using this to produce category-specific embeddings, by providing instructions to focus on each type of similarity in turn. we provide the prompt (see appendix b) to each embedding model at the start of the doc-ument. 10 for claude, we use a modified form of category-specific prompt for summarizing and omit the prompt for the embedding stage. ## results: can existing models perform fine-grained literary sts? we present results using a single general-purpose embedding from each model in truncation and sliding windows show marginal differences. surprisingly, naively truncating to the first segment of the story and taking the mean of sliding window embeddings are similarly success- models overindex on surface features. across all models, spearman's ρ is higher for the four "confounder" categories than any of the fine-grained similarity categories. notably, the author category (which is computed solely from exact match of author ids) has the highest score more frequently than any other category. this indicates that embeddings are much more sensitive to author-specific stylistic factors than to the fine-grained semantic factors captured in the remaining categories. some sensitivity to author and fandom is expected-some authors will focus on different types of stories, and some conventions or styles will be more common in one fandom than another. the overall (fandom-specific) category takes this into account by considering similarity based on both fandom-specific and non-fandom tags. however, 77% of the models in while capturing author and fandom information is not inherently harmful, the outsized impact of these (trivially computable from metadata) features on embedding-based similarity scores limits their applicability to analysis looking for more subtle phenomena like theme or trope similarity. category-specific embeddings show minimal impact. ## related work long-context and embedding evaluation. a number of datasets for long-context evaluation have included literary texts. booksum @xcite involves summarization over public domain books. longbench, longbenchv2, and helmet @xcite @xcite include question answering over narrativeqa @xcite ; @xcite introduces summarization and qa tasks over a set of novels with entity names replaced to reduce the impact of potential contamination. embedding-focused datasets include sts tasks but focus primarily on very short ## conclusion we present ficsim, a dataset of stories and similarity labels for benchmarking model performance on long-context sts tasks within fictional texts. using ficsim, we show that there is no single model that performs well across all types of similarity-and there are types of similarity for which no model performs well. in corpora with strong superficial similarities, like author or fandom overlap, embeddings may capture this information at the expense of other types of similarity. for this specific type of task, bigger (or more expensive) models are not uniformly better than their smaller, cheaper alternatives. our evaluation of sliding window attention and category-specific embeddings also demonstrates that sensible modifications to the model to adapt to long-form or literary texts have a minimal impact on performance. we call for the careful evaluation of models on the particular task they are applied to, with annotation or validation by subjectmatter experts. the poor performance of otherwise strong models on ficsim highlights the substantial gap that exists between current models and their utility for literary applications. our models fail to capture finegrained literary similarity and overindex on superficial features of the text in their embeddings. we expect that clever system design or additional domainspecific training could improve performance within this generation of embedding models, and we encourage the evaluation on literary tasks for future embedding model releases. we hope ficsim will help digital humanities researchers make informed decisions about model selection for tasks relating to story similarity, encourage more evaluation of embedding models on dh tasks, and serve as an example of how creative works can be used for academic research without circumventing creators' rights and wishes.