update readme and plots

README.md

@@ -10,13 +10,20 @@ pipeline_tag: image-segmentation
 widget:
 - src: samples/610160855a90f10006fd303e_10_00418.tif
   example_title: Urban scene
+license: cc
+metrics:
+- accuracy
+- f1
+- iou
 ---
 
 # Model Card for Restor's SegFormer-based TCD models
 
-This is a semantic segmentation model that can delineate tree cover in aerial images.
+This is a semantic segmentation model that can delineate tree cover in high resolution (10 cm/px) aerial images.
 
-This model card refers to several models uploaded to Hugging Face. The model name refers to the specific architecture variant (e.g. nvidia-mit-b0 to nvidia-mit-b5) but the broad details for training and evaluation are identical.
+This model card is mostly the same for all similar models uploaded to Hugging Face. The model name refers to the specific architecture variant (e.g. nvidia-mit-b0 to nvidia-mit-b5) but the broad details for training and evaluation are identical.
+
+This repository is for `tcd-segformer-mit-b0`
 
 ## Model Details
 
@@ -27,13 +34,15 @@ This semantic segmentation model was trained on global aerial imagery and is abl
 - **Developed by:** [Restor](https://restor.eco) / [ETH Zurich](https://ethz.ch)
 - **Funded by:** This project was made possible via a (Google.org impact grant)[https://blog.google/outreach-initiatives/sustainability/restor-helps-anyone-be-part-ecological-restoration/]
 - **Model type:** Semantic segmentation (binary class)
-- **License:** Model training code is provided under an Apache-2 license. NVIDIA has released SegFormer under their own research license. Users should check the terms of this license before deploying.
+- **License:** Model training code is provided under an Apache-2 license. NVIDIA has released SegFormer under their own research license. Users should check the terms of this license before deploying. This model was trained on CC BY-NC imagery.
 - **Finetuned from model:** SegFormer family
 
-### Model Sources [optional]
+SegFormer is a variant of the Pyramid Vision Transformer v2 model, with many identical structural features and a semantic segmentation decode head. Functionally, the architecture is quite similar to a Feature Pyramid Network (FPN) as the output predictions are based on combining features from different stages of the network at different spatial resolutions.
+
+### Model Sources
 
-- **Repository:** Link to Restor repo
-- **Paper [optional]:** TBD
+- **Repository:** https://github.com/restor-foundation/tcd
+- **Paper:** We will release a preprint shortly.
 
 ## Uses
 
@@ -67,22 +76,28 @@ There is no substitute for trying the model on your own data and performing your
 
 ## How to Get Started with the Model
 
-You can see a brief example of inference in [this Colab notebook]().
+You can see a brief example of inference in [this Colab notebook](https://colab.research.google.com/drive/1N_rWko6jzGji3j_ayDR7ngT5lf4P8at_).
 
-For end-to-end usage, we direct users to our prediction and training pipeline which also supports tiled prediction over arbitrarily large images, reporting outputs, etc.
+For end-to-end usage, we direct users to our prediction and training [pipeline](https://github.com/restor-foundation/tcd) which also supports tiled prediction over arbitrarily large images, reporting outputs, etc.
 
 ## Training Details
 
 ### Training Data
 
-The training dataset may be found [here](https://huggingface.co/datasets/restor/tcd), where you can find more details about the collection and annotation procedure. Our image labels are released under a permissive CC-BY-Attribution license.
+The training dataset may be found [here](https://huggingface.co/datasets/restor/tcd), where you can find more details about the collection and annotation procedure. Our image labels are largely released under a CC-BY 4.0 license, with smaller subsets of CC BY-NC and CC BY-SA imagery.
 
 ### Training Procedure
 
-Our training and inference pipeline may be found [here](). We used a 5-fold cross-validation process to adjust hyperparameters during training, before training on the "full" training set and evaluating on a holdout set of images. The model in the main branch of this repository should be considered the release version.
+We used a 5-fold cross-validation process to adjust hyperparameters during training, before training on the "full" training set and evaluating on a holdout set of images. The model in the main branch of this repository should be considered the release version.
 
 We used [Pytorch Lightning](https://lightning.ai/) as our training framework with hyperparameters listed below. The training procedure is straightforward and should be familiar to anyone with experience training deep neural networks.
 
+A typical training command using our pipeline for this model:
+
+```bash
+tcd-train semantic segformer-mit-b0 data.output= ... data.root=/mnt/data/tcd/dataset/holdout data.tile_size=1024
+```
+
 #### Preprocessing
 
 This repository contains a pre-processor configuration that can be used with the model, assuming you use the `transformers` library.
@@ -110,27 +125,32 @@ Note that we do not resize input images (so that the geospatial scale of the sou
 
 You should be able to evaluate the model on a CPU (even up to mit-b5) however you will need a lot of available RAM if you try to infer large tile sizes. In general we find that 1024 px inputs are as large as you want to go, given the fixed size of the output segmentation masks (i.e. it is probably better to perform inference in batched mode at 1024x1024 px than try to predict a single 2048x2048 px image).
 
-All models were trained on a single GPU with 32 GB VRAM (NVIDIA RTX3090) attached to a 32-core machine with 64GB RAM. All but the largest models can be trained in under a day on a machine of this specification.
+All models were trained on a single GPU with 24 GB VRAM (NVIDIA RTX3090) attached to a 32-core machine with 64GB RAM. All but the largest models can be trained in under a day on a machine of this specification. The smallest models take under half a day, while the largest models take just over a day to train.
 
-## Evaluation
+Feedback we've received from users (in the field) is that landowners are often interested in seeing the results of aerial surveys, but data bandwidth is often a prohibiting factor in remote areas. One of our goals was to support this kind of in-field usage, so that users who fly a survey can process results offline and in a reasonable amount of time (i.e. on the order of an hour).
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+## Evaluation
 
-### Testing Data, Factors & Metrics
+We report evaluation results on the OAM-TCD holdout split.
 
-#### Testing Data
+### Testing Data
 
 The training dataset may be found [here](https://huggingface.co/datasets/restor/tcd).
 
 This model (`main` branch) was trained on all `train` images and tested on the `test` (holdout) images.
 
-#### Metrics
+![Training loss](train_loss.png)
+
+### Metrics
 
-TODO
+We report F1, Accuracy and IoU on the holdout dataset, as well as results on a 5-fold cross validation split. Cross validtion is visualised as min/max error bars on the plots below.
 
 ### Results
 
-TODO
+![Validation loss](val_loss.png)
+![IoU](val_jaccard_index.png)
+![Accuracy (foreground)](val_multiclassaccuracy_tree.png)
+![F1 Score](val_multiclassf1score_tree.png)
 
 ## Environmental Impact
 
@@ -146,21 +166,25 @@ This estimate does not take into account time require for experimentation, faile
 
 Efficient inference on CPU is possible for field work, at the expense of inference latency. A typical single-battery drone flight can be processed in minutes.
 
-## Citation [optional]
+## Citation
 
-TODO
+We will provide a preprint version of our paper shortly. In the mean time, please cite as:
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
+```latex
+@unpublished{restortcd,
+  author = "Veitch-Michaelis, Josh and Cottam, Andrew and Schweizer, Daniella Schweizer and Broadbent, Eben N. and Dao, David and Zhang, Ce and Almeyda Zambrano, Angelica and Max, Simeon",
+  title  = "OAM-TCD: A globally diverse dataset of high-resolution tree cover maps",
+  note   = "In prep.",
+  month  = "06",
+  year   = "2024"
+}
+```
 
 ## Model Card Authors
-Josh Veitch-Michaelis, 2024
+Josh Veitch-Michaelis, 2024; on behalf of the dataset authors.
 
 ## Model Card Contact
 
-Please contact josh [at] restor.eco for questions or further information.
+Please contact josh [at] restor.eco for questions or further information.