Use the Metadata, Luke! – An Experimental Joint Metadata Search and N-gram Trend Viewer for Personal Web Archives (2022)

Abstract: Many digital humanists (philologists, historians, sociologists, librarians, the audience for web archives) design their research around metadata (publication date ranges, sources, authors, etc.). However, current major web archives are limited to technical metadata while lacking high quality, descriptive metadata allowing for faceted queries. As researchers often lack the technical skill necessary to enrich existing web archives with descriptive metadata, they increasingly turn to creating personal web archives that contain such metadata, tailored to their research requirements. Software that enable creating such archives without advanced technical skills have gained popularity, however, tools for examination and querying are currently the missing link. We showcase a solution designed to fill this gap.

## introduction the potential of the vast amount of data generated on the world wide web (as a corpus of text) has taken the lead from printed media and is constantly evolving (e.g. the u.s. presidential tweets). for example, sociologists and linguists must use online sources to gain insight into recent trends (temporal, ideological standpoints) in social topics and language as the print media can not keep up the pace and provide real-time updates (e.g. covid-19 news, social media). luckily, the state of major web archives indicates that all important data will likely be preserved. therefore, beyond tackling the difficulties of web harvesting, meeting the needs of researchers has become the most recent goal of archivists @xcite . however, the quantity of data in major archives does not counteract deficiencies and negligence in quality. the aforementioned direction of research requires quality descriptive metadata (publication date, author, column, portal name, keywords, etc.) for specific portals with complete and up-to-date archives, to for example assess source dependent standpoints. unfortunately, major web archives are limited to full-text or url searching, even more advanced ones only include filters on technical metadata (e.g. file format, crawl date), and a rare few such as https://webarchives.ca/ have facets on descriptive metadata. as @xcite , @xcite and @xcite have pointed out, for a more detailed overview of sources, a uniform format for metadata and text would be needed because websites are typically neither consistent nor well-structured. nevertheless, the complexity and time-consuming nature of isolating the useful text, identifying the segments along with metadata (e.g. title, headings, image), preprocessing (e.g. tokenization and parsing) and postprocessing (e.g. filtering, deduplication, curation of metadata) in existing web archives make it an unattainable promise of future ai solutions. this is especially relevant for those without technical skills or sufficient infrastructure foot_0 . fortunately, many workflows (e.g. crawling, text extraction, nlp, plotting) can be conducted sufficiently well with a few clicks on free, easy-to-use software or services that work on uploaded data or prepared data sets (see examples in section 3.). researchers with insufficient funds (e.g. phd students) are therefore limited to creating "personal" web archives using such tools to gian at least preliminary results through exploratory analysis and test their ideas before going further. these attempts usually get stuck at the aggregation and visualisation stages, as available tools and tutorials do not facilitate the freedom to experiment, instead only showcasing basic, and typical workflows. with our tool these scholars can find answers to detailed questions involving metadata based on their gathered data with at most a few line of code needed for the collecting and preprocessing steps. such archives must be created manually and in many cases cannot be published or shared due to conservative legal regulations. this puts pressure on website operators, as everyone is forced to create an archive of their own, the very issue that was intended to be avoided by shared archives such as common crawl that have become trapped in a rivalry of quantity over quality. on top of that, the internet archive does not support downloading and restoring pages foot_2 , and as of yet does not contain a faceted query and visualisation interface for descriptive metadata. in the future, large web archives may catch up and support querying quality uniform descriptive metadata out of the box, but until then scholars must individually experiment with the available tools on a smaller scale. here we find the eternal struggle to strike a balance between precision and recall in a non-standardised open domain. off-the-shelf services are too dependent on the provider. moreover, they are constantly growing and developed, which means that these services will inevitably change over time foot_4 (e.g. shut down or being extended), and become volatile foot_5 . this hinders the reproducibility of obtained results, forcing researchers not to trust such sources, since it is essential to ensure the citation and long-term preservation of research data and final research re- ## tools for a modular feav cycle tools such as trafilatura @xcite serve as good examples, as they are adequate for creating custom, "personal" web archives and have unique features compared to existing major web archives (e.g. extracting descriptive metadata and text in a uniform format with sufficient quality and quantity). in a few lines of python code one can create enough research data for preliminary analysis, and by combining such data sets more complicated advanced research questions can be answered in depth. processing such text with nlp pipelines (e.g. spacy @xcite ) with another few lines of code to extract lemmas and named entities is easy even for non-technical researchers. on the visualisation front there exist a large number of tools and libraries providing quick and elegant solutions (e.g. d3.js @xcite ), often implicitly handling most parameters, however, their functionality either focuses on ease-of-use or modularity, rarely both. their popularity -judging by their download and citation counts -shows that people tend to prefer them. they may also appear as building blocks of some full-fledged compact applications as well. the last remaining part is the aggregation of data, which is either done with predefined workflows limiting its scope or is completely left to the user with all of its complexity. following from the design of such pipelines, if any of these tools are not yielding satisfactory results, they can be changed independently (maintaining the feav cycle) as their output is in simple and standard format. we take these tools as granted, and focus on the missing part, the ground-up design of an ideal trend viewer as the aggregation module for the desired tool chain. ## trend viewer trends the history of n-gram trend viewers starts with the google books n-gram viewer @xcite which has an exceptionally large corpus, but its software is not freely available to try on custom data. another similar example is the microsoft web n-gram services, which has been shut down 6 . the national library of norway also established an elaborate n-gram trend search service @xcite on their collection of books, but the source code foot_7 has not been updated for 7 years and is written in the now deprecated python 2 language. its sqlite database back end can handle 34 billion words from books and newspapers in a remarkable way. the project is still relevant as the icelandic gigaword corpus utilises the code in their recent work @xcite . the most versatile existing solution is shine foot_8 ( jackson et al., 2016) , an experimental faceted search engine and trend viewer tool based on solr in its early stages, however, it has not shown signs of development since 2020. it has similar goals to those of our project (see section 1.), including faceted search over various metadata, however, it fails to cover descriptive metadata other than publication date when it comes to aggregation and visualisation. the web archives for longitudinal knowledge (walk) foot_9 project is a good demonstration of applying shine to facilitate dh research. these examples suggests that there is an emerging trend in the dh and nlp of using descriptive metadata for diachronic corpora (created from e.g. web archives) where available. such data augmented with modern nlp (e.g. sentiment analysis, wikification @xcite ), aggregated and displayed in the context of not just time, but all other accessible factors can help digital humanists focus and gain insight on new previously nonexistent scopes of their research. our aggregation module works independently on the output of existing crawling and nlp tools, and uses an existing visualisation tool, therefore maintaining ease of use and sufficient generality. ## method: redesign from the ground up we encourage future researchers to utilise e.g. trafilatura to create their own research data in standard format (e.g. tei xmls) that can be lemmatised effectively with e.g. spacy to get preprocessed data in decent quality and quantity in almost any language foot_10 . we also provide example scripts, however, for the sake of reproducibility we used an existing data set containing descriptive metadata for our experiments: indig et al. (2020) developed a pilot web archive described as a complete, reliable and versatile representation of archived content (recorded in warc files) for hungarian news portals while maintaining archive content authenticity @xcite . it contains around 1 billion words from about 2.8 million news articles written in the last 20 years composed of roughly 20 different hungarian news portals with a standardised layout and descriptive metadata in tei xml format. the whole archive is available with doi links on the zenodo.org repository foot_11 which is maintained by cern ensuring the reproducibility of further downstream research and making a positive example of legally sharing "personal" web archives for non-profit research purposes. building on the various types of available metadata in the input data set (publication date, author, column, portal name, keywords, etc.), we designed an extensible hybrid query service. on one hand, the retrieved records can be viewed as a list of links to the original pages as in traditional search engines, and the query can be refined with filters on metadata fields. additionally authorised non-profit researchers can view -legally, without copyright infringement -the full article in our standardised html format which contains the original text formatting generated on-the-fly from the stored tei xmls. this allows access even if the original link is broken or its content has changed, which also helps manual work and gathering examples. on the other hand, the features of the retrieved documents can be visualised with different kinds of graphs (e.g. line and bar charts) by setting the appropriate parameters (e.g. the fields for the axes), allowing the user to create custom views of the data filtered by the represented features (see figure 1 .) opening possibilities for non-technical users to express research questions (see section 7. for more examples). all retrieved data and plotted graphs can be downloaded for use e.g. in research papers or additional processing. as an example, bundling such a tool with a research paper can enable readers to cross-check and further examine the presented results and enhance reproducibility and reusability, increasing the impact of the paper. ## technical evaluation and details existing search engine platforms e.g. solr or elas-ticsearch provide a solid foundation, however, their complexity and possible vulnerabilities are prone to hacking attempts and require continuous maintenance. their complexity also discourages external developers from customisation attempts. the same holds for deep learning frameworks, which also require advanced technical skills and machinery. birkenes et al. ( 2015 ) have shown that storing n-grams in sqlite can be fast enough even for databases extending over 34 billion tokens and yield correct results. however, they use separate tables for uni-, bi-and trigrams. our approach was to add padding tokens at the end of text segments and create only a table with 5-grams, as we can represent lower-order n-grams using only the first n tokens, thus creating fewer rows overall. the actual frequencies come from grouping the matching lower-order n-grams (group by statement in sql) and summing the frequencies for each group 12 . so one can easily use wider search patterns e.g. "[first name] [surname] [job title] [organisation]". each n-gram has a frequency and is linked to a document while the metadata (publication date, sources, authors, etc.) for each document -identified by a document id -are stored in a row in a separate table along with the path for the tei xml document on the file system. we distinguish three types of fields: a) simple text (e.g. portal name, column), b) multi-valued text (e.g. author, keywords), c) date (e.g. publish date) which can specified as an interval when used as a filter. our import script requires a conll-u like vertical tsv format used by sketch engine @xcite . the possible lemmas for each word are acquired by lemmatising the text prior to database creation with a common lemmatiser. we only store the lemma n-grams in the database, because their disambiguation in the query string is done by a look-up table using all possibilities in conjunctive normal form (cnf) as ambiguous n-grams are unlikely and results close to real lemmatisation can be achieved. our pipeline uses awk and standard unix tools, allowing to create and count lemma 5-grams for 1 billion words while retaining their source document ids, in about an hour. the user has to write a configuration file and run a single command to import the data into the database. we provide example scripts to help users from the crawling through to the importing stages. to allow rewriting or different development paths, we decoupled the front end and the back end which communicate with a stable rest api. the application is available as a docker image to facilitate operating system independence and long term usage. the only limiting factor we experienced is the quality of the used data, as regarding performance even low-end hardware can handle databases with the the expected size. ## examples search engines for web archives or interfaces for visualising trends usually help researchers navigate through data sets, allowing quick assessment of the usefulness of materials for current research (e.g. cross-check data to reveal artefacts), possibly verifying simpler claims or filtering useful parts (saving time by not processing unused segments). we would like to illustrate how, thanks to more detailed preprocessing, a few tiny adjustments can expand the possibilities for answering more complex research questions. we provide presets in the webui for users to show the examples given below. one can observe the emergence and rivalry of new terms across different news portals, e.g. in relation to the climate crisis foot_13 , or the way how medical jargon foot_14 became common language during the covid-19 pandemic @xcite , and how the correlation between its key issues (e.g. restriction, quarantine vs. vaccination) have changed over time by the start of the third wave. another example demonstrates how one can get a quick answer to the question "which values (national, chatolic, european) are more prominent in the articles of the selected news portals?", which can then be nuanced through more detailed examination (e.g. their ideological standpoints). a small but important detail is that distortions due to the size of each portal can be mitigated by expressing the occurrences of n-grams as a percentage (e.g. percentage of articles per portal). when listing relevant articles the logical operators applied to the n-gram can be used to separate the trend of co-occurrence and independent occurrences of the listed n-grams. thanks to accurate metadata extraction, the use of hashtags by online newspapers and journalists can be tested, the variety and consistency of keywords and phrases associated with topics can be a telling feature, and it helps to improve the accuracy of the search and the weighting of the topic. in the field of sports, we can investigate the interest of some hungarian news portals in hungary and abroad, by querying articles about different sports (see figure 1 ). we would expect that portals in hungary would be predominantly concerned with football results and in romania hockey (jégkorong) would be more prominent (e.g. on maszol.ro). the fact that in hungarian three synonyms of almost equal rank are typically used to describe football (labdarúgás, futball, foci) makes interpretation difficult, because synonyms sometimes complement each other and sometimes overlap, which is reflected in the number of results. this can be clarified with the downloadable tsv files. future plans include making it even easier to do this directly in the search engine by utilising vector space models to unify synonyms if enabled. a further point to note is the importance of context. if researchers want to quantify traces of hate speech or ethnic exclusion, they need to be fa-miliar with the background and the attitudes of the sources used. a simple but striking example is the occurrence of the term gypsy (cigány). in two of the newspapers studied, the results are very prominent, but one is an avowedly extremist medium (far-right wing), while the other deals specifically with social issues, education and inclusion. ## conclusion and discussion we introduced a search and visualisation engine for viewing n-gram trends in light of different descriptive metadata implemented in a lightweight sql-based framework. a custom data set can be imported with a few lines of python code, and the customised visualisation can be exported to be used directly in research papers. this helps researchers express their questions freely or share their data through a query and visualisation interface (e.g. as anonymous supplementary material for research papers) and enable further examination and possible new discoveries on the same data set. we plan to add more nlp modules (e.g. sentiment analysis to study hate speech and wikification for semantic results) for different layers of annotation and support for cql expressions. the source code 15 and docker image is published under a copyleft license and a public pilot service is available.