update readme and plots

README.md

@@ -8,201 +8,183 @@ datasets:
 - restor/tcd
 pipeline_tag: image-segmentation
 widget:
-  - src: samples/610160855a90f10006fd303e_10_00418.tif
-    example_title: Urban scene
+- src: samples/610160855a90f10006fd303e_10_00418.tif
+  example_title: Urban scene
+license: cc
+metrics:
+- accuracy
+- f1
+- iou
 ---
 
-# Model Card for Model ID
+# Model Card for Restor's SegFormer-based TCD models
 
-<!-- Provide a quick summary of what the model is/does. -->
+This is a semantic segmentation model that can delineate tree cover in high resolution (10 cm/px) aerial images.
 
+This model card is mostly the same for all similar models uploaded to Hugging Face. The model name refers to the specific architecture variant (e.g. nvidia-mit-b0 to nvidia-mit-b5) but the broad details for training and evaluation are identical.
 
+This repository is for `tcd-segformer-mit-b3`
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
+This semantic segmentation model was trained on global aerial imagery and is able to accurately delineate tree cover in similar images. The model does not detect individual trees, but provides a per-pixel classification of tree/no-tree.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+- **Developed by:** [Restor](https://restor.eco) / [ETH Zurich](https://ethz.ch)
+- **Funded by:** This project was made possible via a (Google.org impact grant)[https://blog.google/outreach-initiatives/sustainability/restor-helps-anyone-be-part-ecological-restoration/]
+- **Model type:** Semantic segmentation (binary class)
+- **License:** Model training code is provided under an Apache-2 license. NVIDIA has released SegFormer under their own research license. Users should check the terms of this license before deploying. This model was trained on CC BY-NC imagery.
+- **Finetuned from model:** SegFormer family
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+SegFormer is a variant of the Pyramid Vision Transformer v2 model, with many identical structural features and a semantic segmentation decode head. Functionally, the architecture is quite similar to a Feature Pyramid Network (FPN) as the output predictions are based on combining features from different stages of the network at different spatial resolutions.
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/restor-foundation/tcd
+- **Paper:** We will release a preprint shortly.
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+The primary use-case for this model is asessing canopy cover from aerial images (i.e. percentage of study area that is covered by tree canopy).
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This model is suitable for inference on a single image tile. For performing predictions on large orthomosaics, a higher level framework is required to manage tiling source imagery and stitching predictions. Our repository provides a comprehensive reference implementation of such a pipeline and has been tested on extremely large images (country-scale).
 
-### Downstream Use [optional]
+The model will give you predictions for an entire image. In most cases users will want to predict cover for a specific region of the image, for example a study plot or some other geographic boundary. If you predict tree cover in an image you should perform some kind of region-of-interest analysis on the results. Our linked pipeline repository supports shapefile-based region analysis.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Out-of-Scope Use
 
-[More Information Needed]
+While we trained the model on globally diverse imagery, some ecological biomes are under-represented in the training dataset and performance may vary. We therefore encourage users to experiment with their own imagery before using the model for any sort of mission-critical use.
 
-### Out-of-Scope Use
+The model was trained on imagery at a resolution of 10 cm/px. You may be able to get good predictions at other geospatial resolutions, but the results may not be reliable. In particular the model is essentially looking for "things that look like trees" and this is highly resolution dependent. If you want to routinely predict images at a higher or lower resolution, you should fine-tune this model on your own or a resampled version of the training dataset.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+The model does not predict biomass, canopy height or other derived information. It only predicts the likelihood that some pixel is covered by tree canopy.
 
-[More Information Needed]
+As-is, the model is not suitable for carbon credit estimation.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+The main limitation of this model is false positives over objects that look like, or could be confused as, trees. For example large bushes, shrubs or ground cover that looks like tree canopy.
 
-[More Information Needed]
+The dataset used to train this model was annotated by non-experts. We believe that this is a reasonable trade-off given the size of the dataset and the results on independent test data, as well as empirical evaluation during operational use at Restor on partner data. However, there are almost certainly incorrect labels in the dataset and this may translate into incorrect predictions or other biases in model output. We have observed that the models tend to "disagree" with training data in a way that is probably correct (i.e. the aggregate statistics of the labels are good) and we are working to re-evaluate all training data to remove spurious labels.
 
-### Recommendations
+We provide cross-validation results to give a robust estimate of prediction performance, as well as results on independent imagery (i.e. images the model has never seen) so users can make their own assessments. We do not provide any guarantees on accuracy and users should perform their own independent testing for any kind of "mission critical" or production use.
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+There is no substitute for trying the model on your own data and performing your own evaluation; we strongly encourage experimentation!
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
+You can see a brief example of inference in [this Colab notebook](https://colab.research.google.com/drive/1N_rWko6jzGji3j_ayDR7ngT5lf4P8at_).
 
-[More Information Needed]
+For end-to-end usage, we direct users to our prediction and training [pipeline](https://github.com/restor-foundation/tcd) which also supports tiled prediction over arbitrarily large images, reporting outputs, etc.
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The training dataset may be found [here](https://huggingface.co/datasets/restor/tcd), where you can find more details about the collection and annotation procedure. Our image labels are largely released under a CC-BY 4.0 license, with smaller subsets of CC BY-NC and CC BY-SA imagery.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
+We used a 5-fold cross-validation process to adjust hyperparameters during training, before training on the "full" training set and evaluating on a holdout set of images. The model in the main branch of this repository should be considered the release version.
 
+We used [Pytorch Lightning](https://lightning.ai/) as our training framework with hyperparameters listed below. The training procedure is straightforward and should be familiar to anyone with experience training deep neural networks.
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+A typical training command using our pipeline for this model:
 
-#### Speeds, Sizes, Times [optional]
+```bash
+tcd-train semantic segformer-mit-b3 data.output= ... data.root=/mnt/data/tcd/dataset/holdout data.tile_size=1024
+```
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Preprocessing
 
-[More Information Needed]
-
-## Evaluation
+This repository contains a pre-processor configuration that can be used with the model, assuming you use the `transformers` library.
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+You can load this preprocessor easily by using e.g.
 
-### Testing Data, Factors & Metrics
+```python
+from transformers import AutoImageProcessor
+processor = AutoImageProcessor.from_pretrained('restor/tcd-segformer-mit-b3')
+```
 
-#### Testing Data
+Note that we do not resize input images (so that the geospatial scale of the source image is respected) and we assume that normalisation is performed in this processing step and not as a dataset transform.
 
-<!-- This should link to a Dataset Card if possible. -->
+#### Training Hyperparameters
 
-[More Information Needed]
+- Image size: 1024 px square
+- Learning rate: initially 1e4-1e5
+- Learning rate schedule: reduce on plateau
+- Optimizer: AdamW
+- Augmentation: random crop to 1024x1024, arbitrary rotation, flips, colour adjustments
+- Number of epochs: 75 during cross-validation to ensure convergence; 50 for final models
+- Normalisation: Imagenet statistics
 
-#### Factors
+#### Speeds, Sizes, Times
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+You should be able to evaluate the model on a CPU (even up to mit-b5) however you will need a lot of available RAM if you try to infer large tile sizes. In general we find that 1024 px inputs are as large as you want to go, given the fixed size of the output segmentation masks (i.e. it is probably better to perform inference in batched mode at 1024x1024 px than try to predict a single 2048x2048 px image).
 
-[More Information Needed]
+All models were trained on a single GPU with 24 GB VRAM (NVIDIA RTX3090) attached to a 32-core machine with 64GB RAM. All but the largest models can be trained in under a day on a machine of this specification. The smallest models take under half a day, while the largest models take just over a day to train.
 
-#### Metrics
+Feedback we've received from users (in the field) is that landowners are often interested in seeing the results of aerial surveys, but data bandwidth is often a prohibiting factor in remote areas. One of our goals was to support this kind of in-field usage, so that users who fly a survey can process results offline and in a reasonable amount of time (i.e. on the order of an hour).
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+## Evaluation
 
-[More Information Needed]
+We report evaluation results on the OAM-TCD holdout split.
 
-### Results
+### Testing Data
 
-[More Information Needed]
+The training dataset may be found [here](https://huggingface.co/datasets/restor/tcd).
 
-#### Summary
+This model (`main` branch) was trained on all `train` images and tested on the `test` (holdout) images.
 
+![Training loss](train_loss.png)
 
+### Metrics
 
-## Model Examination [optional]
+We report F1, Accuracy and IoU on the holdout dataset, as well as results on a 5-fold cross validation split. Cross validtion is visualised as min/max error bars on the plots below.
 
-<!-- Relevant interpretability work for the model goes here -->
+### Results
 
-[More Information Needed]
+![Validation loss](val_loss.png)
+![IoU](val_jaccard_index.png)
+![Accuracy (foreground)](val_multiclassaccuracy_tree.png)
+![F1 Score](val_multiclassf1score_tree.png)
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+This estimate is the maximum (in terms of training time) for the SegFormer family of models presented here. Smaller models, such as `mit-b0` train in less than half a day.
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** NVIDIA RTX3090
+- **Hours used:** < 36
+- **Carbon Emitted:** 5.44 kg CO2 equivalent per model
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+Carbon emissions were be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-## Technical Specifications [optional]
+This estimate does not take into account time require for experimentation, failed training runs, etc. For example since we used cross-validation, each model actually required approximately 6x this estimate - one run for each fold, plus the final run.
 
-### Model Architecture and Objective
+Efficient inference on CPU is possible for field work, at the expense of inference latency. A typical single-battery drone flight can be processed in minutes.
 
-[More Information Needed]
+## Citation
 
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+We will provide a preprint version of our paper shortly. In the mean time, please cite as:
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
+```latex
+@unpublished{restortcd,
+  author = "Veitch-Michaelis, Josh and Cottam, Andrew and Schweizer, Daniella Schweizer and Broadbent, Eben N. and Dao, David and Zhang, Ce and Almeyda Zambrano, Angelica and Max, Simeon",
+  title  = "OAM-TCD: A globally diverse dataset of high-resolution tree cover maps",
+  note   = "In prep.",
+  month  = "06",
+  year   = "2024"
+}
+```
 
-[More Information Needed]
+## Model Card Authors
+Josh Veitch-Michaelis, 2024; on behalf of the dataset authors.
 
 ## Model Card Contact
 
-[More Information Needed]
+Please contact josh [at] restor.eco for questions or further information.