Great Bibliographies as a Source of Data for the Humanities – NLP in the Analysis of Gender of Book Authors in G erman Countries and in P oland (1801-2021) (2023)

Abstract: The subject of this article is the application of NLP and text-mining methods to the analysis of two large bibliographies: Polish one, based on the catalogs of the National Library in Warsaw, and the other German one, created by Deutsche Nationalbibliothek. The data in both collections are stored in MARC 21 format, allowing the selection of relevant fields that are used for further processing (basically author, title, and date). The volume of the Polish corpus (after filtering out non-relevant or incomplete items) includes 1.4 mln of records, and that of the German corpus 7.5 mln records. The time span of both bibliographies extends from 1801 to 2021. The aim of the study is to compare the gender distribution of book authors in Polish and German databases over more than two centuries. The proportions of male and female authors since 1801 were calculated automatically, and NLP methods such as document vector embedding based on deep BERT networks were used to extract topics from titles. The gender of the Polish authors was recognized based on the morphology of the first names, and that of the German authors based on a predefined list. The study found that the proportion of female authors has been steadily increasing both in Poland and in German countries (currently around 43%). However, the topics of women’s and men’s writings invariably remain different since 1801.

## introduction the research conducted straddles two broad areas. one relates to the issue of the resources and methodologies (nlp and text-mining tools applied to large bibliographies -hereafter lb), and the other concerns a certain problem that belongs to the field of cultural anthropology and/or social sciences (equal gender participation in various social activities). we will first address the latter issue, while the former (data and methods) will be presented in the subsequent sections. the complex and long-standing processes leading to a fairer participation of both genders in social, scientific, economic, and cultural life have been ongoing in europe for at least two centuries. their most visible public representation was for decades the feminist movement initiated in the usa in the mid-19 th century (women's convention in seneca falls, usa, 1848). the actual gender status in european countries, however, is different in specific regions and largely independent of official policies and spectacular events publicized by the media. the real scope of the ongoing changes in this area can only be revealed by analyzing big data that consistently and synthetically reflect the state of affairs over a long period of time. in particular, a convincing analysis of this phenomenon should not highlight the most mediated, individual exponents of women's status (e.g. awards, prominent positions in politics or business). it should rather rely on information resources aggregating large amounts of data dispersed in various sources that we can consider the most objective possible. large bibliographies can be considered as sources that meet these conditions. they consist of data which represent a very important segment of intellectual life and are collected systematically according to the same principles. also of significance is the fact that for decades, lb have remained insufficiently exploited by scientific methods. the present work is therefore of a pioneering nature. it should be noted that, so far, in empirical studies of complex processes related to gender equality, economic and legal measures have been used. however, they can hardly be trusted in the case of a politically fragile region with a turbulent history, such as central europe. its multilingual diversity, the intensity of political change, different monetary systems, and border shifts over the past two centuries make publication stream analysis a more reliable approach that is highly resistant to the influence of random factors or systemic disruptions. writing books is, among other things, the result of one's prosperity, exposure to knowledge, and education. in addition, social acceptance of the author's gender and social origin was necessary for the publication of a book. for example, medicine in the nineteenth century was almost entirely masculinized. therefore, a female author of a work on, for example, surgery would not have been accepted by the readers, ergo no publisher would publish such a work. ## research objective and hypothesis the first goal of this study is to show the proportions of women and men among the authors of books published from 1801 to 2021 in two european communities of communication, associated in various periods with the concept of a nation or of a state. for that purpose, we generated historical histograms showing the proportions of male, female, and unrecognized authors. a chronological (vertical) survey allowed us to show the dynamics of cultural change, occurring in the most populous and influential countries of central europe over a long period of time. the second goal of the research was to extract topics specific to male and female authors from book titles. its aim was to identify the main contents or topics that characterize the writing of both genders. we assumed that the authorship of a book aggregates a number of socially and psychologically salient variables that are difficult to capture, especially over a long period of time. topics were generated from time sections, representing main historical periods. we hypothesized that in the 19 th century (until ca 1910) there would be little, if any, similarity between the two sets both in polish and in german data. we also expected to observe a growing overlap of topics generated from titles by male and female authors after the interwar period 1918-1945 for both polish and german data. the question as to whether this parameter is likely to become a universal and effective measure of gender equality remains open. yet, it certainly provides a quantitative and possibly objective estimate of this phenomenon. its advantage, from the point of view of methodology, is the use of big and "clean" data and the ensuing independence from random factors. another issue was the choice of poland and germany as the objects of the study. these are countries geographically and culturally close but historically different in size and status. despite the instability of borders and political systems, german culture maintained continuity in the 19 th and 20 th centuries based on various state organisms (including the rhine union, the second reich, the weimar republic, the third reich, and the federal republic of germany). additionally, in the 19 th century german countries (especially prussia) were among the world leaders in science and culture. the situation of poland was radically different. poland lost its statehood at the end of the 18 th century, and in effect existed until world war i only as an entity identified with its culture, history, religion, and, above all, its language. and even this status of an "imagined state" was constantly challenged by the occupation regimes. the country regained its independence in 1918, but then again was occupied between 1939 and 1945 by the third reich and the soviet union. after 1945, poland became a vassal state subordinated to the soviet union; not regaining full sovereignty until 1989. given the above circumstances, the discovery of similarities (or differences) related to the gender of book authors in both corpora will provide a result that seems objective, and scientifically relevant. ## related work much more research is now focused on the exploratory analysis of library catalogs around the world, for example, @xcite ; @xcite . however, surprisingly little research has been published on the use of artificial intelligence techniques in bibliography data @xcite @xcite . the problem of topic recognition from short texts is studied in the literature @xcite , but application of such a method to a large number of book titles, that is, very short texts, is an original approach. moreover, as discussed in section 5 the applied method differs from methods known from literature @xcite . moreover, the paper deals with the problem of merging two bibliographies and deduplication of records (see section 4). the problem is discussed in the literature @xcite @xcite . ## research material the study was carried out on large bibliographies produced by the national libraries of poland 1 and germany (deutsche nationalbibliothek) foot_1 . although these are not the official "national bibliographies", they functionally fulfill the conditions set for such monumental repositories. in particular, they have a predictable structure of data, the permanent care of a central institution, and the aspiration of covering the entire body of writings. it is worth mentioning that due to the lack of a polish state in the 19 th century @xcite , there was no central institution to keep track of publications during this period. records extracted from the estreicher polish bibliography foot_2 , which registers polish works and polonica from a period very poorly represented in the collection of the national library, were therefore taken into account. for this purpose, the online part of the bibliography was used, which covers approximately 40% of the entire collection. the records of the central libraries are stored in marc format @xcite which allows for field searches and elimination of works that do not meet the analysis conditions. in the first iteration, records lacking authors or titles were filtered out. all nontext works (maps, notes, gramophone records, other sound recordings, etc.) were omitted. periodical publications were also discarded. in the case of works extracted from the estreicher polish bibliography, 5 pages were accepted as the lower limit of acceptable volume (this source includes very short documents too). in principle, most works in languages other than polish or german were eliminated from both bibliographies. however, this criterion was problematic, as there is no easy way to automatically distinguish works by german (or polish) authors writing in other languages from authors of other nationalities, but publishing in germany or poland (a frequent example from the 20 th century are german doctorates in english). several works by polish authors from the 19 th century that for political reasons were published in the languages of states that occupied polish territory (mainly in german or russian) or in some other other international language (e.g., latin, french, and ruthenian), were excerpted by hand and included. the whole research material for german comprised 25.9 mln records, out of which 72% were rejected as nonrelevant, while for polish a total of 2.3 mln records were processed (95% from the polish national library database, 5% from the estre-icher bibliography), and 38% were rejected (38% among those from the national library database, 50% from the estreicher bibliography). the distribution of data over time was not balanced, but this does not impede the results of our study (with incomplete representation, inductive inference is applicable). the association of first names with gender is almost unambiguous and, in addition, the feminine gender in polish is always indicated by the name ending -a. in the case of the polish base, recognition was, therefore, based on this rule. the automatically generated database (mapping of name to gender) was then manually checked. some names, especially foreign ones, are ambiguous, so they were marked as unknown. the german language, on the other hand, is not so consistent, as some old high german names end in a consonant (e.g., annetrud, edeltraut, gudrun etc.). an additional difficulty -especially after 1945 -in german, is the large number of borrowed names. for this reason, a reference catalog of german male and female names was prepared using open resources. it was manually verified again and completed manually for missing names that had an occurrence larger than 20. finally, the gender of the authors was determined by automatically comparing their names with the list. ## methods of data processing the marc database is structured, but often requires additional procedures for information retrieval. for example, the publication date often contains additional characters that need to be removed. the author's first name is not a separate field and has to be automatically separated from the last name. another problem was the use of umlaut character encoding (the umlaut was encoded as two characters) in german data that are not compliant with standard utf-8. for processing marc-21 files we have used a python 3 pymarc foot_3 library. in the case of the estreicher polish bibliography, the data are available as html tables, and the date of publication is an element of a single text containing the place of publication and the name of the publisher. this required the development of a set of heuristic rules to extract the publication data from the text. another problem with the estreicher bibliography was the need for deduplication with the national library of poland. a special method was developed that first splits data based on publication date and then uses jaccard's token similarity between titles and authors to detect duplicates in a group of papers with the same publication date. detecting topics from short texts requires a dedicated procedure @xcite , which differs from classic approaches to topic modeling, such as lda @xcite . this is because titles are very short and we cannot rely on the co-occurrence of words in the same text and on the assumption that the text is a mixture of several topics. therefore, the semantic analysis of titles (the second objective of the work) was based on a clustering procedure. this procedure requires a measure of similarity between titles. therefore, each title was transformed into a vector space using deep bert networks @xcite . since pre-trained bert networks are not suitable for solving semantic similarity problems @xcite , we used a sentence-bert approach @xcite based on metric learning @xcite . it uses siamese and triplet network structures to derive semantically meaningful sentence embeddings. in the case of polish, we tuned the sentence-bert network on a publicly available corpus foot_4 of human-annotated sentence pairs in polish for their semantic relatedness starting from the herbert @xcite pretrained model. in the case of german, we used an already predeveloped sentence-bert model dedicated to english and german text foot_5 . having embedded text, we applied classical k-means @xcite clustering to obtain what we hoped were clusters of semantically distinct groups of titles. we then described the clusters using a set of words that identifies the content of the clusters. this is done using a modified tf-idf @xcite procedure that takes into account class information (c-tf-idf), which was proposed in @xcite . this method yields list of words with their probabilities for each cluster; that is, topic as a concept used in topic modeling approaches @xcite . our approach differs from bertopic @xcite by omitting umap (mcinnes et al., 2020) used for document vector reduction and replacing hdb-scan (mcinnes and healy, 2017) clustering with k-means. the authors tested bertopic, but the results were not satisfactory because hdbscan's outlier detection function caused about 75% of the data to go off-topic, and umap's preservation of only local similarities caused semantically different titles to be mixed within a single topic. detected topics can be linked to gender by counting the number of titles assigned to a topic/cluster with authors of that gender. in the case of multiauthor books, we required that each author have the same gender. to express the relevance of a topic to a given gender, the importance index is defined as the ratio of books authored by women to men (or vice versa) about a given topic (cluster) normalized to the ratio of books for each gender. this allows the detection of a topic relevant to both men and women, irrelevant of the gender distribution among authors. ## conclusions the research presented here was developed by combining advanced nlp techniques, mathematical statistics, programming, and large bibliographic data. it has demonstrated that the ratio of male and female authors in book publishing, when measured over a long period of time, should be considered one of the most reliable indicators of women's empowerment in the society. in the context of politically unstable regions, it has two main advantages: it is relatively time-proof in the period of late modernity (i.e. approximately since 1800), and it is synthetic. the former characteristic implies that the data are comparable over a long period of time. they were created under similar conditions (open publishing market) and the object of measurement remains the same (books from the 19 th and the 20 th century do not differ in essence). the latter feature means that it includes some specific measures that economics, history, or cultural anthropology used to apply separately (access to education, financial standing, social status, etc.). it also showed to be sensitive to one-time events such as wars, political or technological breakthroughs. the study confirmed the hypothesis that in german countries and in poland similar upward trend in gender equality may be observed (figure 2 ). however, the question remains open as to whether this phenomenon would have a similar dynamics throughout europe. the current share of the authors of the book is approximately 43% women and 57% men (note that these are values after deducting unrecognized items, so slightly different from those in figure 2 ). the value of the crosscorrelation coefficient, i.e., 0.93, confirmed that statistically the two processes (gender equality in german countries and in poland) may be observed are not identical, although very similar. an interesting issue is whether participation of women in public life was higher in poland or in german countries. the overall ratio of female authors in poland and germany is slightly higher in poland (12.1%) than in german countries (11.1%). analyzing figure 2 , one can also ask whether there is a target state of optimal social balance between both genders. for example, would the ideal be an equal share of male and female authors? perfect sym-metries are a product of human imagination and expectations, rather than empirical observable phenomena @xcite . equal parities should be treated with distrust in the social sphere as well as attempts to realize new utopias, not different in essence from those once conceived by philosophers, e.g. thomas morus (utopia) or tommaso campanella (city of the sun). differences in the psychological profiles and interests of men and women have always existed and -as we demonstrate in our study -translate into various types of book content published. therefore, a balanced and socially favorable level of participation of both genders among book authors would have to be considered 50% with a large margin, even 10%. it seems that this point will soon be reached both in germany and in poland. the result of the research on the topics confirms some of the above statements. multiple analyses of the entire corpus, as well as of its horizontal sections (19 th and 20 th centuries, contemporary period), conducted on german and polish data, confirmed that the areas of interest of male and female authors are different. their thematic profiles, generated using machine learning methods (bert language models), shows a wide number of almost non-shared topics. this result does not, of course, resolve the issue of gender, and thus whether it should be seen as a purely biological or culturally conditioned phenomenon. however, it is an important contribution to the discussion on this topic, as it is based on sound methodology and a massive factual resource from two languages and cultures. the concluding remark applies to all the research conducted here. it shows that the analysis of large bibliographies by methods of data science, text mining, corpus linguistics, and nlp is a new, fully-fledged, promising strand of research.