Zero-shot Probing of Pretrained Language Models for Geography Knowledge (2023)

Abstract: Gauging the knowledge of Pretrained Language Models (PLMs) about facts in niche domains is an important step towards making them better in those domains. In this paper, we aim at evaluating multiple PLMs for their knowledge about world Geography. We contribute (i) a sufficiently sized dataset of masked Geography sentences to probe PLMs on masked token prediction and generation tasks, (ii) benchmark the performance of multiple PLMs on the dataset. We also provide a detailed analysis of the performance of the PLMs on different Geography facts.

## introduction transformer based pretrained language models (plms) have proven to be effective on multiple tasks in nlp ranging from the standard information extraction and text classification to more complex ones such as reading comprehension and text generation. multiple such transformer based plms are available, either trained from scratch on large amounts of data or fine-tuned for specific tasks and domains. it is also being established @xcite that on multiple nlp tasks, plms with billions of parameters (llms) such as gpt-4, bloom and opt, perform better than plms with significantly lesser number of parameters ('small plms') such as bert and roberta. as plms are being widely used in multiple applications, their performance needs to be improved either by rigorous methods such as full scale fine-tuning or through efficient methods such as prompt based few-shot fine-tuning @xcite , adapters @xcite and low rank adaptation (lora) @xcite . in this paper we attempt to gauge the performance of multiple plms on facts pertaining to geography. the facts we check include information pertaining to three types of geographical entities -natural (rivers, mountain ranges, natural reserves, etc.), geo-political (countries, cities, etc.) and public/industrial facilities (dams, power plants, amusement parks, etc.). we hypothesize that plms may not be trained well on such niche geography knowledge and efforts must be invested to enrich this aspect of their learning. this evaluation exercise is the first step towards such an enrichment effort. to build a corpus of such geography facts from text, we obtain geography facts in the form of triples from wikidata and use templates to arrange them as masked sentences (prompts) to probe the plms (section 2). to gauge the quality of the developed prompts, we carry out a manual examination of randomly sampled sets of prompts and check for triviality, grammatical incorrectness and noise (section 2.2). as the first contribution, we release this benchmark dataset foot_0 of 5268 masked sentences pertaining to various aspects of world geography, which can be used for probing and fine-tuning exercises. we consider a host of plms and probe them on the created masked sentences and report the comparative performance. we present an analysis of the behaviour of different plms on the different kinds of geography facts we probe. we also present which of the considered fact types are easiest or hardest for the plms to answer. this analysis forms our second contribution (section 4) of benchmarking the performance of multiple small plms on this task, thereby suggesting application designers of knowledge systems to consider the reported analysis. ## dataset creation fact type example wikidata property (propp) example triple (entitys, propp, entityo) geo-political country in which a certain city is located. london is located in england country (p17) (london, p17, england) continent in which a certain country is located japan is located in asia. continent (p30) (japan, p30, asia) capital of a certain country tokyo is the capital of japan. capital (p36) (japan, p36, tokyo) natural countries which are basin countries to a certain sea rivers from greece and turkey flow into the aegean sea. basin country (p205) (aegean sea, p205, greece), (aegean sea, p205, turkey) highest point of a mountain range mount everest is the highest point of the himalayan mountain range. highest point (p610) (mount everest, p610, the himalayas) waterbody which has created a canyon the grand canyon is created by the colorado river. located in or next to a body of water (p206) (grand canyon, p206, colorado river) public and industrial facilities waterbody on which a certain dam is located the aswan dam is located on the nile river. located in or next to a body of water (p206) (aswan dam, p206, the nile) country in which a certain power station is located the turbigo power station is located in italy. country (p17) (turbigo power station, p17, italy) country in which a certain amusement park is located the wonderland amusement park is located in china. country (p17) (wonderland amusement park, p17, china) we collect instances of 23 different types of geographical facts which we would like to test the plms for and categorize them under three heads. as part of the head -natural, facts pertaining to natural entities namely sea, mountain range, forest, river, desert, waterfall, canyon and natural reserve are considered. as part of the head -geo-political, facts pertaining to geo-politically relevant entities namely continent, country, city, air base and naval base are considered. as part of the head -public/industrial facilities, facts pertaining to entities relevant to public life (work and leisure), namely dam, power station, mine, amusement park and stadium are considered. the different fact types considered with their examples are shown in ## probing pre-trained language models we aim to evaluate plms for their geography knowledge. given our constraints of using license friendly and less resource consuming models, we consider the following set of language models -encoders: bert (large-cased), roberta (largecased), albert (large-uncased) and distilbert (uncased); decoders: gpt-neo (2.7b), falcon (7b), falcon-instruct (7b) and mpt (7b), lead-ing us to a total of 8 different plms to probe and evaluate. we currently do not report on the encoderdecoder models such as flan-t5 and bart as their preliminary results are poor and hence require more investigation. we describe in brief (i) the plms considered in the exercise and (ii) our preliminary experiments with encoder-decoder models, in appendix d. ## evaluation and analysis it is desired that the token predicted with highest probability for the mask token's place should be correct, indicating the learning of the plm to be complete for that fact. similarly for generative lms, the token predicted right after the input text we can also consider a softer representation based similarity between the predicted tokens and the desired ones instead of exact match to handle variations such as us, usa and america. however, that would inevitably bring into play, some form of thresholds on the similarity score, which would be difficult to guess without training. we believe that the top-k/near-k evaluation metric also helps handle this aspect. ## note on other plm probing benchmarks an interesting research direction is gauging whether llms can replace knowledge graphs and latest work such as @xcite conclude that such replacement is far from reality. this calls for increasing research focus towards making llms more knowledgeable both generally and domain-wise. in the current context, it hence becomes important to highlight the need for a specific geography focused plm probing dataset when there are several plm probing benchmarks available in the literature @xcite @xcite . firstly, to the best of our knowledge this is the first only geography focused plm probing benchmark. secondly, we believe that the existing ones cover a wide variety of general facts and information and hence for this focused domain, they would cover only a limited set of facts. specifically, we discuss one of the foremost ones -the lama benchmark @xcite . the lama benchmark considers four sources to build their probing benchmark out of which only the conceptnet and trex sources consist of concepts related to geography or spatial. a closer exploration of the conceptnet source reveals that the lama authors include a "atlocation" relation but the facts checked are too general, such as "something you find at the [@xmath0.". this is different from the current goal of discovering geography knowledge. the trex source comes closer to our method and considers a set of wikidata based relations including a few geography ones. however, the trex's procedure maps a given wikidata triple to multiple sentences from wikipedia text (wiki text) sentences. lama's procedure selects one of the multiple candidates randomly for probing, which may include other facts about the entities in the triples not necessarily relevant to geography. e.g., entities egypt and africa occurring in nongeographical context as in the trex sentence -the song's lyrics of unity mention a number of countries, including england, russia, china, egypt and israel, as well as the continent of [@xmath1. moreover, probing plms, which have seen wikipedia text as part of their training data, on masked sentences made form wikipedia text itself might give them an advantage as compared to our template-based generation of masked sentences which would be different structure wise from the training data, leading to a more effective probing exercise. another closely related benchmark dataset is the geoglue @xcite , which also presents a set of evaluation tasks to gauge geographic language understanding, but is primarily in chinese. other relevant literature focused on spatial and geography knowledge exploration in text though focused largely towards question answering is discussed in @xcite , @xcite and @xcite . ## limitations (and future work) we are aware about the facets and avenues that the current exercise doesn't consider and they remain to be explored in detail. a few important ones form part of the future work are listed as follows: penchant for hardware poor: currently we do not include very large models such as the 13/40 billion or even larger models due to our goal of exploring resource poor and license friendly plms. this allows us better reach, deployment and usecase wise. however, we do plan to include larger models in the benchmarking exercise. fine-tuning: the focus on using smaller models also prompts us to improve the performance through different plm fine-tuning techniques. a comprehensive fine-tuning exercise is underway and will be separately discussed. deeper understanding: investigation into the attention patterns of the lms' transformer blocks might be necessary to gain deeper insight into what conspires when geography prompts are seen by the lm. we plan to employ attentionflows @xcite and @xcite in this regard. better templates: currently the templates are encoder model friendly as we started with these models and are catching up with the more recent llms. this urges us to design better templates which can work seamlessly for both encoder and generative models. ## conclusion and future work we aimed at evaluating the learning of pre-trained language models in the space of geography knowledge. to carry out the evaluation we created a probing dataset of 5268 masked sentences based on wikidata triples. using the masked token prediction and text generation tasks, we probe 8 different plms (4 encoders and 4 decoders) and report the results. we observe that encoder models such as bert showcase relatively better knowledge of geography facts than the generative models considered. we elaborate the results through various analyses and examples of fact-types and prompts where the plms perform well and otherwise. catherine yeh, yida chen, aoyu wu, cynthia chen, fernanda viégas, and martin wattenberg. 2023. attentionviz: a global view of transformer attention. arxiv preprint arxiv:2305.03210.