Restructuring and visualising dialect dictionary data: Report on E rzya and M oksha materials (2025)

Abstract: There are a number of Uralic dialect dictionaries based on fieldwork documentation of individual minority languages from the Pre-Soviet Era. The first of these published by the Finno-Ugrian Society features the Mordvin languages, Erzya and Moksha.In this article, we describe the possibility of reusing XML dialect dictionary collection point and phonetic variant data for visualizing informative linguistic isoglosses with R programming language’s Shiny web application frame-work.We provide a description of the ‘H. Paasonen Mordvin Dictionary’, which will possibly provide the reader with a better perspective of what data and challenges might present themselves in minority language dialect dictionaries.We provide a description of how we processed our data, and then we provide conclusions followed by a more extensive section on limitations. The conclusions state that only some of the data should be rendered with R Shiny web application, whereas some data might be better rendered by other applications.Our limitations section description calls for the extension the dialect dictionary database for a more concise description of the languageforms.

## introduction the dictionaries of endangered languages are very valuable in contemporary research. many dictionaries, however, are not available digitally, and if they are, they may not have ocr accuracy that would make them fully searchable. the mere size and extent of dictionaries in large majority languages can make them challenging to process. especially the work done with the transkribus platform @xcite has made high quality text recognition available to an exceptionally large community. at the same time, the successful recognition of diacritical marks has opened many new avenues for further work on texts written using finno-ugric transcription, as reported by @xcite . the field is clearly moving toward the point where many dialect dictionaries will become digitally available. dialect dictionaries, however, present a relatively complicated data type, as the internal data structures are not always easily retrievable from the printed text, especially if we do not have all the formatting. part of what contributes to this challenge is that the traditional dictionaries contain many different types of data: various derivations, figure 1 : example of an embedded word article in h. paasonens mordwinisches wörterbuch. band ii (k-m) @xcite compounds, dialectal variants and example sentences, all appearing with various components of geographic data. the example in figure 1 illustrates how the forms of the word fledermaus 'bat' are presented inside a larger macro-article, and the geographic locations are presented with abbreviations. this data is very well structured and detailed, but it is organized for a printed dictionary. when this data can be rendered in new ways, disconnected from the original layout of the printed pages, many new research questions and methods begin to appear. data visualizations and interactive applications are often seen in the digital humanities, which, in many ways, are elementary for understanding the structures of more complex datasets. in this study, we introduce methods and application we have developed to visualize and inspect geographically coded erzya and moksha dictionary data. the example application is built in the r language's (r core team, 2021) shiny framework @xcite , and is hosted on csc -it center for science's rahti service. eventually we plan to host the application and store the data in the language bank of finland. shiny is a framework for building and testing interactive web applications that can execute r code. in our opinion, shiny is a very suitable tool for rapid prototyping, but we do acknowledge that different approaches should be investigated for long term deployment. our application uses leaflet javascript library through r's leaflet package @xcite and datatables javascript library through r's dt package @xcite , which all generate javascript, but the visualization is controlled through interactive r session within shiny framework. there is some unnecessary overhead in this solution, as the same could be achieved with javascript alone. yet, as the application in itself is fairly simple and very easy to maintain in the current form, this framework serves the current needs very well. all code is openly available in github, with the documentation of our hosting solutions and most up-to-date url. 1 the working model we have developed and present here connects especially to situations where we have the original dictionary as some kind of a digital file that contains the original formatting, or where the original formatting can be retrieved one way or another. this differs drastically from situations where the dictionary data is available as a database or within some software regularly used in dictionary compilation task. however, our situation is very realistic, as many printed dictionaries can be found in formats such as digital print files, text documents of some type, or we may have a version where text is retrieved through text recognition and, ideally, proofread carefully. if the data already exists in a database or other digital structure, importing it to our application would also be a trivial task. ## the erzya and moksa dialect dictionary the erzya and moksha dialect materials used in this dictionary represent fieldwork collections organized or performed by heikki paasonen at the end of the 1800s and beginning of the 1900s. geographically, the fieldwork was extensive, representing over 200 collection points for the two languages combined. there is an inconsistency, however, in the representation of language materials from the various collection points, i.e., whereas there are 10,737 phonetically documented word forms found for the erzya village of marisevo, on the one hand, there are only four word forms attributed to the erzya village of kabaevo (see rueter, 2016, 134), on the other. similar figures can be presented for moksha, too. it may also be noted that the geographic granularity of moksha-language collection represents a different level of polygons, i.e., erzya materials appear to have more village-level representation, whereas their moksha counterparts might be more readily associated with a volost' or raion-level representation. despite these shortcomings, the 'mordvin dialect dictionary' is, in fact, the most extensive documentation of erzya and moksha vocabulary published. the materials come from the mordwinisches wörterbuch 'mordvin @xcite @xcite @xcite @xcite based on the heikki paasonen works and collections @xcite @xcite @xcite @xcite @xcite @xcite . the dictionary data were originally fed into a desktop in the 1980s and 1990s, and the resulting materials were converted into an xml unicode document based on style, size and font parameters. even though there was a high consistency in the usage of fonts for distinguishing what nowadays could be handled with unicode ranges, some of the same problems that confound us today also occurred, namely, language abbreviations such as erzya (e) and moksha (m) and other look-alikes frequently required correction before the different languages and dictionary structure abbreviation data were clean. the 2,703-page dictionary consists of 6,952 macro articles, each of which represents a distinct word root. the macro articles can be divided further into 21,754 stem word entries, which range in complexity from a single-stem article with russian and german translations to a macro article containing multiple-stem articles with additional compound-word articles and etymologies. in more complex articles, it becomes apparent that a stem word article can distinguish three separate sections where collection point data are mentioned. these sections are phonetic variants of a given cognate, the definitions, which may vary from place to place, and example contexts. occa-sionally, semantic cross-referencing is made where a word from on collection point may be associated with an entirely different word from another collection point. etymological references indicate cognates in other languages, although a majority of the uralic cognates or parallel forms were left out of the final version of the printed dictionary. ## related work we contextualize our work within the cartography of the uralic languages, and geographic visualization of the language documentation data and traditional fieldwork based data collected in early modern times. we do not position our work strongly toward dialect geography or research of geographic variation, primarily as we mainly have worked on visualization of the collected or already published data, but do not address how this data would be further used in research on these dialects and their variation. naturally, the same datasets that are used in these applications can be also used in a wide array of different research purposes. the primary purpose of our application is to allow visual inspection of the data, helping to understand underlying geographic distributions and what kind of possible gaps or other structures there are. naturally, the application may be extended in the future to allow more complicated tasks. one problem with visualizing dialect data is that tools intended for semi-professional or professional cartographers, although powerful, are very specialized, have a high learning curve and are heavy to run. at the same time, the data we process when comparing dialectal variants is relatively simple. @xcite reviewed a number of light and practical programs that could be used in this task, and we believe their suggestions and observations are relevant also today. another wide survey of visualization platforms was presented by @xcite . in the context of uralic languages, the recent cartographic work by @xcite has been very important, as they have produced openly licensed maps about the distribution of the uralic languages. as they primarily operate with polygon level, expressing the language areas, our work is very complementary to theirs. as a hypothesis, the collection points of the paasonen's dictionary data should fall within the traditional erzya and moksha areas as shown in the dataset of @xcite . indeed, it seems obvious that with rapidly changing technology we will be using new tools of visualization and analysis each decade. however, the dialectal data in itself remains valuable, even increasing in value as the data can be extended with other resources that deepen the geographic and temporal coverage. from this point of view the visualization is in all ways secondary, and the underlying data the key element. ## data structure we restructure the dialect dictionary entries so that each entry in the derived structure contains only individual word forms and their dialectal variants. we then introduce literary-language lemmas to plot the entries as individual items on a map. we separate the management of lexical data and coordinates, so that the 'location' connects the lexemes with their coordinates. this allows that the coordinate data can be stored in a separate table or in other format that is independent from the lexical data and does not need to be modified in several places at once. similar structure was used also by @xcite . we use columns 'base_form', 'variant', 'location' and 'language'. to manage the lexical data. additional column that will need to be added when the materials are combined from various sources is 'source'. the erzya and moksha data could at later point be appended with contemporary dialect data, and the source for this information would then differ from paasonen's. the data about the source will be stored in an additional table, as it contains information about the collection time, authors and correct citations. at the moment the application contains data only from one source, so the references can be stored at a higher level. with the column 'base_form' we are currently rather free on what kind of content should be placed there. it is not possible to decide on one base lexeme that would match for both erzya and moksha, but especially when the visualization contains data from just one language, this seems like an easiest alternative. for the erzya and moksha application we also have added numbers for each lexeme, but we do not believe this is the best solution going forward. one possible approach is to use as the base form a descriptive translation that would then also be used to select the current lexeme for viewing. in the original structure of the paasonen's digitized dictionary, derivation articles are child articles of a single macro article, and compound words are addressed in grandchild articles. although various parts are connected to one another, they are related to different semantical lexical items, and compound word items may be mentioned as grandchild articles under each component macro article. the relations between root words (the first child article of the macro article), derivations (nonfirst child articles of the macro article), compound words and multiword contexts (grandchild articles of the macro article) are retained in the nested xml structure, so the original structure can, if needed, be retrieved. we start by modifying the xml structure of the digitized dictionary where the layout has been converted to tagged elements. 2 this situation is very specific to the dictionary presently under inspection, and it does not necessarily serve as a model for further work. the data is read and rendered as a tabular structure where one row is one word form from one location. this structure is versatile for cases where there are different amounts of data from different locations for different words. ## application we have currently set up two application prototypes. one visualizes the erzya and moksha data, and another serves as an editor for the data in our structure. 3 the editor is very much at the preliminary testing stage, but it allows uploading and downloading the files that can be edited also locally. as shown in figure 2 , the application interface contains multiple elements. these are described below. the application has a selection part on upper left corner where the user can browse the entries according to their german translations. under that basic information about the word form is presented. all variants attested are displayed on an interactive map in the middle. the map is fully interactive, and when a dot is clicked, we see the name of the location, attested dialectal word form and the language (erzya or moksha). under the map there is a table that displays all data rows. the table can be searched and filtered. the example sentences connected to the entry are not currently displayed, but this could be added at a later stage. they are often attested for indi-vidual locations, but we see it currently as an open question how to best display them. one possibility one could be to show all of them below other interface components. each entry with the same phonetic representation is coded with the same color on the map. this makes it easy to see different patterns and compare these word for word. the colors are currently selected automatically from a predefined palette. occasionally, the same collection point may have more than one dialectal variant. here it was important that we apply a jitter function to both latitude and longitude readings that moves all points a bit randomly, so overlying data can be displayed. this does not seem to cause loss of information at the scale where this data operates. with locations very close to one another the impact of jitter should be monitored and checked, but the currently used values are effective for the data at hand. another approach, tested by our collaborator cinthia ishida (federal university of pará), would be to overlay different shapes in these situations. ## conclusion in the future the application will be extended for use with other dictionaries, especially for the uralic languages, for which the dialect dictionaries have been created within the same research tradition. at the same time we are participating in a collaboration between researchers of the uralic languages and the languages of the amazonian region. this will allow for more extensive testing and will possibly necessitate adjustments for some additional information present in them. at the same time we aim to keep the structure simple enough so that different dictionary types can be readily used as data sources. it is not our goal, however, to visualize all the possible information in the original dictionary in one application. instead, we envision that some of the same data might be transformed in various ways and displayed in applications that are more suitable for the aspects one is interested in. however, overlaying various different data types or displaying several maps side by side would be one feature that is so central for the usability of the application, that we may integrate this functionality very rapidly. at the same time our flexible data model makes it easy to reuse the same data in other novel environments as needed.