Paper: GeoLLM: Extracting Geospatial Knowledge from Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Assuming the covariates are indeed available, does the current method offer complimentary benefits to further improve the performance?

Author response:
**A3:** We added an additional baseline “Nighlight” in Table 1 and Table 3. Please refer to “Stronger baselines” in our general response for the description and discussion of this baseline. The Nightlight baseline provides insights into the baseline performance attainable using a well-established satellite image based method. Notably, LLMs seem to be far more robust on a wide variety of tasks when compared to this baseline. Somewhat predictably, Nightlight is only particularly strong on the population tasks. This supports the idea that LLMs are a richer source of knowledge and a better covariate compared to the well-established nightlight satellite imagery. In practice, GeoLLM, satellite imagery, and other covariates would be used together for the best predictions.

Reviewer comment (Experiments):
The authors, if possible, should also make comparisons with fine-tuning several other geo-spatial LLM type models, like K2 (Deng et. al. 2023) through fine-tuning on the same set of data.

Author response:
**A4:** The main contribution of the paper is to present a novel method to extract geospatial knowledge from LLMs. It would be very interesting to determine whether or not the additional domain specific pre-training and finetuning of geospatial LLMs such as K-2 embeds additional knowledge that can be leveraged by GeoLLM. However, we believe that this investigation would be orthogonal from the main purpose of GeoLLM, which was simply to show that LLMs are new covariates and to present a method to use them as such. We leave this investigation to future work.

Reviewer comment (Reproducibility):
Lack of clear prompt templates and answer pairs design. The paper mentions the use of multiple data sources and various tasks; however, there are significant deficiencies in the theoretical approach. Specifically, the paper does not clearly specify whether the prompt formats are the same for each task and lacks detailed descriptions of how prompt templates and answer pairs were designed for different tasks. This lack of clarity makes it challenging for readers to understand the experimental design and poses obstacles for replication and further research.

Author response:
**A1:** We added Appendix A.1 to provide details on the prompt template that we used across all tasks as well as an additional prompt example that we use in the asset wealth task. You can find details for the source of the auxiliary data used to construct the prompts in section 3.2. We hope this aids in the replicability of our results.

Reviewer comment (Reproducibility):
Lack of clear experimental design details. Furthermore, in terms of experimental design, there are shortcomings in this paper. It does not provide a clear overview of the experimental training process, including the selection of data, preprocessing steps, and hyperparameter tuning. This omission makes it difficult to evaluate the detailed experimental procedure, hindering the reproducibility of the experiments.

Author response:
**A2:** We added Appendix A.2 to describe the finetuning process and the exact hyperparameters used. Details on the preprocessing of the data for prompts are provided in the last paragraph of section 3.1. Details on the ground truth data for tasks, such as their sources or any task-specific preprocessing that we do, are provided in section 4.1.

Reviewer comment (Evaluation):
Absence of comparison with ground-truth. Additionally, the paper does not thoroughly explore the comparison between experimental results and real-world geospatial data (Ground Truth). Accuracy of geospatial knowledge is crucial in practical applications; however, the paper lacks comparative analysis with actual geographical data.

Author response:
**A3:** Pearson’s $r^2$, which is the correlation squared, is shown for every model across all tasks in table 1. These are calculated using the ground truth and the predictions. We also show the mean absolute errors in an identically structured table in Appendix A.3. We clarify the fine-tuning process of the LLMs in Appendix A.2 where we describe how we use unsupervised learning on the prompts and labels (ground truth) concatenated. We also try to use the word “labels” more to be more clear about the fine-tuning process and the fact that we use ground truth. Please let us know if this is a sufficient clarification or if we are misunderstanding your point.

Reviewer comment (Experiments):
The scope of experiments could be further expanded to support the conclusion. Please consider the Questions below for details.

Author response:
With the revision, we add a sentence embeddings baseline, satellite imagery baseline, results for GPT-2 with GeoLLM, 3 new ablations, few-shot performance with GPT-3.5 and GPT-4, and a preliminary evaluation of the biases of LLMs. Please see “Zero-shot and few-shot experiments”, “Full set of ablations”, and “Preliminary investigation of bias” in the overall response.

Reviewer comment (Evaluation):
While Pearson's r^2 is used for comparisons with previous literature, can I see more comparisons with baselines on other evaluation metrics like MAE or MSE?

Author response:
As a potentially more interpretable metric, we present the mean absolute error (MAE) for all models across all tasks and training sample sizes in Table 3 in Appendix A.3. One can observe that the same conclusions that are made with Pearson's $r^2$ can be made when comparing the MAE of the various models. In particular, the relative ranking in performance of the models within tasks is consistent with Pearson's $r^2$.

Reviewer comment (Evaluation):
Could the paper present the label distribution to evaluate performance variations between urban and rural areas, or densely versus sparsely populated regions?

Author response:
While there does not appear to be any obvious signs of performance bias across countries or continents as seen in fig. 2, the existence of biases in performance is inevitable as the internet training corpora of LLMs are inherently biased towards developed and densely populated areas. We show preliminary signs of this bias in table 5 in Appendix A.5. We find that there is a moderate increase in mean absolute error between densely populated and sparsely populated areas and another moderate increase between the areas with above median asset wealth and below median asset wealth. As we discussed in section 5, this demonstrates that GeoLLM has the potential to be used as a tool to reveal LLM biases on a geographical scale. However, substantial additional research is required for a comprehensive analysis of the geospatial biases in LLMs and their training corpora.

Reviewer comment (Evaluation):
The paper does not provide a detailed analysis of the potential biases of LLMs and their training corpora.

Author response:
**A1:** While there does not appear to be any obvious signs of performance bias across countries or continents as seen in fig. 2, the existence of biases in performance is inevitable as the internet training corpora of LLMs are inherently biased towards developed and densely populated areas. We show preliminary signs of this bias in Table 5 in Appendix A.5. We find that there is a moderate increase in mean absolute error between densely populated and sparsely populated areas and another moderate increase between the areas with above median asset wealth and below median asset wealth. As we discussed in section 5, this demonstrates that GeoLLM has the potential to be used as a tool to reveal LLM biases on a geographical scale. However, substantial additional research is required for a comprehensive analysis of the geospatial biases in LLMs and their training corpora.

Reviewer comment (Experiments):
It would be better to compare the GeoLLM’s performance with the results from satellite images.

Author response:
**A2:** Please see the Nightlight baseline in the “Stronger baselines” section of the overall response.

Reviewer comment (Experiments):
Have you tried the zero-shot or few-shot performance of various LLMs on the presented tasks?

Author response:
**A3:** Please see “Zero-shot and few-shot experiments” in the overall response.

Reviewer comment (Experiments):
It seems that the proposed GeoLLM can only perform one specific task. After fine-tuning, is the fine-tuned LLMs (e.g., GPT-3.5) able to retain the ability to answer general questions that is not related to the specific task? Including some discussions about the generalization part could be useful.

Author response:
**A1:** GeoLLM is not intended to retain natural language or general-purpose response capabilities. It is a method to introduce a novel set of covariates that can enhance various downstream geospatial ML tasks that currently depend on traditional covariates such as satellite images. These tasks include poverty estimation, food security, biodiversity preservation, and environmental conservation. Thus, we can quickly adapt powerful LLMs to a variety of geospatial tasks using our novel prompting strategy. Moreover, we demonstrate that a high level of performance is achievable with relatively cheap finetuning using methods such as QLoRA.

Reviewer comment (Experiments):
The proposed model can only handle static/tabular geo information. It does not handle other types of spatial data, or spatiotemporal data and tasks.

Author response:
**A2:** As LLMs have not yet been shown to be effective covariates for geospatial ML tasks, we focus on providing a method that enables this by efficiently extracting their spatial knowledge. In fact, there are two general settings for spatial prediction: location modeling and spatial contextual modeling [1]. The former aims at predicting the attributes/characteristics of a place given its current location. The latter focuses on predicting the attributes/characteristics of a place given its location as well as its spatial context. Our GeoLLM is able to handle both types of tasks. The spatial context modeling corresponds to our current GeoLLM setting since we give the location and description of the current location as well as its nearby places extracted from OpenStreetMap (See Figure 1 (b) bottom). The location modeling corresponds to the setting in which we delete the “Nearby place” in our prompt. We believe we are the first one that shows LLMs’ abilities on a wide range of geo-tasks. We acknowledge that there are other geo-tasks such as spatial relation prediction between two places, or questions requiring spatial footprints more than points (e.g., polylines and polygons). We treat them as future work.

Reviewer comment (Reproducibility):
Is there any specific tokenization process introduced for GPS coordinates? Normally, the direct tokenizers with LLMs could split the GPS point into several different tokens, which might undermine the ability to understand GPS coordinates correctly.

Author response:
**A5:** We recognize the tokenization problem of the current LLM for geo-coordinates.  Current LLMs break up the GPS coordinates into multiple tokens and makes understanding them more difficult, especially for smaller LLMs like GPT-2. While larger models such as GPT-3.5 are better able to utilize the numeric value of the coordinates, as shown in the “only coordinates'' ablation in Table 2, the coordinates do not affect the final performance of GPT-3.5 significantly (bump from 0.72 to 0.73 Pearson’s $r^2$) as shown in the “no coordinates'' ablation in table 2.  In fact, GeoLLM is motivated by this observation and is developed to overcome this limitation. How to develop a tokenizer that makes LLM better understand geo-coordinate is a very interesting research direction but is orthogonal to our GeoLLM framework. We leave it as one of our future work.

Reviewer comment (Experiments):
Why using classification setting? The current loss functions that LLMs are using is a kind of binary, where only predictions that exactly match the masked word are considered correct and rewarded, while all other predictions are considered incorrect and penalized. So, there is no has no awareness and no sense of being close to or far away the correct answer. Normally, for the task described in this paper, this kind of sense should be valuable in getting better performance.

Author response:
**A6:** Making geospatial predictions via the generation of tokens is significantly more difficult than regression. As one would expect, GPT-2 struggles to perform well within this paradigm, even doing much worse than MLP-BERT that uses regression. For this very reason, we deliberately introduced decimals in the labels (0.0 to 9.9 as opposed to 0 to 99) to add consistency to the predictions as they always require exactly 3 tokens to generate (e.g. “5”, “.”, “3”). However, to our surprise, the autoregressive nature of LLMs not seem to hinder GPT-3.5 and Llama 2 significantly, with GPT-3.5 exceeding the performance of all models that use regression, including RoBERTa and the newly introduced MLP-BERT and Nightlight baselines. In addition, as GPT-3.5 only allows for fine-tuning through OpenAI’s API, we cannot make any modifications to the tokenization or architecture of the best performing model by far.

Reviewer comment (Evaluation):
What about the deployment cost of the proposed GeoLLM? For example, compared to the baselines, what are the fine-tuning costs and inference costs of using GPT-3.5 or other LLMs. From my experience, fine-tuning GPT-3.5 can be very expensive.

Author response:
**A7:** We present the costs of the finetuning of the LLMs we use in Appendix A.2. In short, fine-tuning GPT-3.5 with 10,000 samples costs 33 dollars. Thanks to its sample efficiency, one can finetune it with 1,000 samples for 5 dollars without much sacrifice to the overall performance. As a set of predictions only needs to be made once or a small number of times for a specific application, the deployment cost is not as significant or as much of a burden as with general-purpose LLMs.