LANTERN-RD: Enabling Deep Learning for Mitigation of the Invasive Spotted Lanternfly
Abstract
The Spotted Lanternfly (SLF) is an invasive planthopper that threatens the local biodiversity and agricultural economy of regions such as the Northeastern United
States and Japan.
As researchers scramble to study the insect, there is a great potential for computer vision tasks such as detection, pose estimation, and accurate identification to have important downstream implications in containing the SLF.
However, there is currently no publicly available dataset for training such AI models.
To enable computer vision applications and motivate advancements to challenge the invasive SLF problem, we propose LANTERN-RD, the first curated image dataset 1 of the spotted lanternfly and its look-alikes, featuring images with varied lighting conditions, diverse backgrounds, and subjects in assorted poses.
A VGG16based baseline CNN validates the potential of this dataset for stimulating fresh computer vision applications to accelerate invasive SLF research.
Additionally, we implement the trained model in a simple mobile classification application in order to directly empower responsible public mitigation efforts.
The overarching mission of this work is to introduce a novel SLF image dataset and release a classification framework that enables computer vision applications, boosting studies surrounding the invasive SLF and assisting in minimizing its agricultural and economic damage.
Introduction
Lycorma Delicatula , dubbed the Spotted Lanternfly (SLF), has recently spread invasively from parts of Eastern Asia to several other continents, establishing successfully in certain new environments due to reasons such as climate preference [9, 10, 25] and the abundance of sustainable plant hosts [1, 13, 12, 26].
Current regions of interest include the American Northeast and Japan, among others [27].
As a new contender in the ecosystem and a nuisance pest, the SLF poses a significant threat to local agriculture and the economy through its destruction of fruit trees, ornamental trees, timber, vineyards and building structures [16, 18, 7], potentially leading to billions of dollars of damage.
Evolving studies propose several methods for containment, from biocontrol agents [2, 4, 15] to trapping mechanisms [8, 19, 6].
Notably, local environmental authorities have mandated quarantines to slow the spread of the SLF [22], and are calling on the public to trap, report, and even take steps to exterminate sighted SLFs [21].
The expanding scope of the invasive SLF problem provides ground for computer vision applications to enhance research studies and containment efforts.
For example, SLF pose estimation may boost understanding of key population-growth activities such as egg-laying and enable motion tracking to characterize population spread [17, 28].
Additionally, while ecologists studying the spatial distribution of the SLF have long relied on manual surveying methods in order to collect data [11, 5, 1, 18], advancements in AI animal detection techniques [20, 29] may improve the efficiency of such data
[Figure]
collection efforts. Finally, the presence of insect species that are visually similar to the SLF (Figure 1) raises additional complications for both researchers and the public, as misidentification of such insects affects the quality of collected data and results in unnecessary harm to native wildlife.
For this issue, there is significant potential for AI-based classifiers to assist ecologists and promote public management efforts [3].
To enable such advancements in computer vision research for the invasive spotted lanternfly problem, there is a great need for a high-quality public dataset of SLF images.
To solve this problem, we present in this work a framework that consists of: (i) LANTERN-RD , a novel image dataset of the invasive SLF and visually-similar insects in order to enable computer vision research for SLF mitigation, (ii) a corresponding baseline convolutional neural network (CNN), and (iii) a mobile deployment of the classifier to empower rapid identification of the SLF against look-alikes and advocate for responsible mitigation efforts.
Dataset
For the purpose of empowering the development of deep learning models to assist in efforts to contain the invasive SLF, we assemble an image dataset of the spotted lanternfly and its look-alikes, as summarized in Figure 1.
2.1 Data Overview
The first iteration of LANTERN-RD includes a total of 5 insect classes: L. delicatula , A. figurata , P. imperialis , U. ornatrix , and H. lineata , with 1187, 1501, 672, 520, and 1970 images, respectively.
The distribution is visualized in Figure 2, and the total size of the dataset is 5850 images, with each image containing exactly one insect.
Sample images of the dataset are displayed in Figure 1.
The data are diverse, including images of insects in different poses, assorted backgrounds, and varied lighting conditions, allowing for the training of generalized models and the generation of sub-datasets for specific tasks.
The dataset is presented as several files of image URLs, grouped by class, alongside a corresponding document outlining labels.
2.2 Pipeline
In order to curate a high-quality dataset, we employ a robust pipeline to gather, clean, and compile data.
Extraction.
Raw image data are sourced from Bing Images and LAION [23, 24].
A number of strategic queries are utilized to capture a large volume of image URLs, which are stored separately according to class.
However, the collected data are noisy and must be filtered before proceeding.
Automated Filtration.
In the first step of cleaning, we use several automated techniques to eliminate extraneous data.
We begin with image hashing to remove duplicate and near-duplicate images.
Next, when image captions are available (as with the LAION data), we parse these captions and eliminate data containing phrases linked to insects other than the queried class.
Through this process, we are able to remove a large volume of noisy data.
Additional Filtration.
We parse the remaining data via several methods, removing the remaining extraneous images.
For example, “Tiger Moth" also happens to be the name of a popular 20th century biplane, and images of this aircraft were highly represented in the raw data before additional cleaning.
We also carefully inspect each image, applying established identification methods to verify that the insect pictured belongs to the label and ensuring that no personally identifiable information remains available.
The filtration process leaves a collection of image URLs that constitute the final, cleaned dataset.
This modular pipeline allows for the efficient integration of new data sources such as community image submissions, enables expansion to additional classes of look-alike species, and takes strides to ensure that the data are clean and useful for training.
Baseline Experiments
Alongside the dataset, we train a baseline CNN using the VGG16 [14] model architecture (under the
MIT License).
The data are randomly split for training, validation, and testing, with a ratio of 60%, 20%, and 20%, respectively.
Before training, the image data are augmented.
Each class pictured in Figure 1 receives a numerical label, and the CNN is trained according to the categorical cross entropy loss function for 50 epochs at a rate of 1e-3.
We train on Tesla K80 GPUs within Google
Colaboratory.
3.1 Results
The model achieved an overall test accuracy of 97.20%.
The per-class F1 scores are summarized in
Table 1.
Mobile Implementation
We additionally propose a simple mobile application that implements the classifier trained in Section 3.
Sensationalization of the spotted lanternfly in affected areas has resulted in the public becoming
[Table]
more closely involved in management efforts, with programs such as New Jersey’s
“Stomp It Out!" [21] calling on citizens to exterminate sighted SLFs.
As a result, it is imperative that the public is able to accurately identify this invasive insect against look-alike species, particularly to prevent well-intentioned citizen scientists from unnecessarily harming wildlife.
Features.
The app allows users to directly capture or upload an image of an insect suspected to be an
SLF.
Subsequently, the picture is fed as an input to the classifier, and the user is notified of the class prediction.
All user data are kept private and operations are run locally.
This app presents one useful application of LANTERN-RD in training deep learning models to assist in efforts to contain the invasive SLF.
We hope that such an application motivates further advancements from the computer vision community.
Conclusion
In this paper, we have introduced LANTERN-RD, a curated dataset consisting of diverse images of the invasive spotted lanternfly and visually similar insects.
This dataset contains 5850 images of the spotted lanternfly and four visually similar insects, and is curated via an efficient pipeline that is scalable to additional data sources and new classes.
A baseline classifier trained on LANTERN-RD achieves a 97.20% test accuracy.
This validates that datasets such as LANTERN-RD will enable a wide array of computer vision applications that have positive downstream impacts on efforts to contain the invasive spotted lanternfly.
To explore one avenue of computer vision applications for the invasive SLF problem, we implement the preliminary classifier into a simple app designed for users to use their mobile devices to rapidly understand whether or not an insect is the SLF.
This assists in research activities and boosts caution on behalf of citizen scientists.
We warn against abuse of the dataset and model; researchers are urged to heed evolving guidelines on spotted lanternfly mitigation.
Future work would include solicitation of community image submissions in order to expand the scale of the dataset, particularly in new and underserved insect classes, and for different stages of the SLF lifecycle.
We call on ecologists and the computer vision community to come together in applying LANTERN-RD for deep learning tasks poised to increase the efficiency, reliability, and scale of efforts to contain the invasive SLF
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Limitations discussed throughout the work and particularly in Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Though we design this work for positive ecological impact, we recognize and indicate negative impacts in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The raw dataset is openly-accessible.
This is work-in-progress research and additional models and data will be made open-source to the public for reproducibility and research.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Training details specified for benchmark model in Section 3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
We propose a benchmarked dataset in this work and do not conduct traditional experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 2. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Section(c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] We discuss in Section 2 the pipeline used to remove such data.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]