xhagrg commited on
Commit
0c5a963
1 Parent(s): 1bc1d73

Update Readme.

Browse files
Files changed (1) hide show
  1. README.md +20 -6
README.md CHANGED
@@ -8,17 +8,16 @@ tags:
8
  ---
9
 
10
  ### Model and Inputs
11
- Prithvi is a first-of-its-kind temporal Vision transformer pre-trained by the IBM and NASA team on contiguous US Harmonised Landsat Sentinel 2 (HLS) data. Particularly, the model adopts a self-supervised encoder developed with a ViT architecture and Masked AutoEncoder learning strategy with an MSE loss function. The model includes spatial attention across multiple patches and also temporal attention for each patch.
12
 
13
  ![](GFM.png)
14
 
15
- The model expects remote sensing data in a video format (B, C, T, H, W). Note that the temporal dimension is very important here and not present in most
16
- other works around remote sensing modeling. Being able to handle a time series of remote sensing images can benefit a variety of downstream tasks. The model can also handle static images, which can be simply fed into the model with T=1.
17
 
18
  ### Pre-training
19
- The model was pre-trained with NASA's HLS2 L30 product (30m granularity) from the contiguous United States. The bands that were used are the following:
20
 
21
- 1. Blue
22
  2. Green
23
  3. Red
24
  4. Narrow NIR
@@ -26,7 +25,7 @@ The model was pre-trained with NASA's HLS2 L30 product (30m granularity) from th
26
  6. SWIR 2
27
 
28
  ### Code
29
- The model follows the [original mae repo](https://github.com/facebookresearch/mae) with some modifications including:
30
 
31
  1. replace 2D patch embed with 3D patch embed;
32
  2. replace 2D positional embed with 3D positional embed;
@@ -42,3 +41,18 @@ python Prithvi_run_inference.py --data_files t1.tif t2.tif t3.tif --yaml_file_pa
42
 
43
  ### Finetuning examples
44
  Examples of finetuning the model for image segmentation using the mmsegmentation library are available through Hugging Face (e.g. [burn scars segmentation](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-burn-scar), [flood mapping](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-sen1floods11), and [multi temporal crop classification](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-multi-temporal-crop-classification)), with the code used for the experiments available on [github](https://github.com/NASA-IMPACT/hls-foundation-os/tree/main/fine-tuning-examples). This also contains instructions to finetune the model for flood detection on the popular open access [sen1floods11 dataset](https://github.com/cloudtostreet/Sen1Floods11).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ---
9
 
10
  ### Model and Inputs
11
+ Prithvi is a first-of-its-kind temporal Vision transformer pre-trained by the IBM and NASA team on contiguous US Harmonised Landsat Sentinel 2 (HLS) data. The model adopts a self-supervised encoder developed with a ViT architecture and Masked AutoEncoder (MAE) learning strategy, with an MSE loss function. The model includes spatial attention across multiple patches and also temporal attention for each patch.
12
 
13
  ![](GFM.png)
14
 
15
+ The model accepts remote sensing data in a video format (B, C, T, H, W). Note that the temporal dimension (T) is very important in this application and not present in most other works around remote sensing modeling. The ability to handle a time series of remote sensing images can benefit a variety of downstream tasks (e.g. Burn Scars segmentation, Flood Segmentation, Land Cover Classification). The model can also handle static imagery which can be fed into the model with T=1.
 
16
 
17
  ### Pre-training
18
+ The model was pre-trained with NASA's HLS V2 L30 product (30m granularity) from the contiguous United States. The bands that were used are the following:
19
 
20
+ 1. Blue
21
  2. Green
22
  3. Red
23
  4. Narrow NIR
 
25
  6. SWIR 2
26
 
27
  ### Code
28
+ The model follows the [original MAE repo](https://github.com/facebookresearch/mae) with some modifications including:
29
 
30
  1. replace 2D patch embed with 3D patch embed;
31
  2. replace 2D positional embed with 3D positional embed;
 
41
 
42
  ### Finetuning examples
43
  Examples of finetuning the model for image segmentation using the mmsegmentation library are available through Hugging Face (e.g. [burn scars segmentation](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-burn-scar), [flood mapping](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-sen1floods11), and [multi temporal crop classification](https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M-multi-temporal-crop-classification)), with the code used for the experiments available on [github](https://github.com/NASA-IMPACT/hls-foundation-os/tree/main/fine-tuning-examples). This also contains instructions to finetune the model for flood detection on the popular open access [sen1floods11 dataset](https://github.com/cloudtostreet/Sen1Floods11).
44
+
45
+ ### Citation
46
+
47
+ If this model helped your research, please cite `Prithvi-100M` in your publications. Here is an example BibTeX entry:
48
+
49
+ ```
50
+ @misc{Prithvi-100M,
51
+ author = {Jakubik, Johannes and Chu, Linsong and Fraccaro, Paolo and Bangalore, Ranjini and Lambhate, Devyani and Das, Kamal and Oliveira Borges, Dario and Kimura, Daiki and Simumba, Naomi and Szwarcman, Daniela and Muszynski, Michal and Weldemariam, Kommy and Zadrozny, Bianca and Ganti, Raghu and Costa, Carlos and Watson, Campbell and Mukkavilli, Karthik and Roy, Sujit and Phillips, Christopher and Ankur, Kumar and Ramasubramanian, Muthukumaran and Gurung, Iksha and Ji, Wei and Avery, Ryan and Ramachandran, Rahul and Maskey, Manil and Olofossen, Pontus and Fancher, Elizabeth and Lee, Tsengdar and Murphy, Kevin and Duffy, Dan and Little, Mike and Alemohammad, Hamed and Cecil, Michael and Li, Steve and Khallaghi, Sam and Godwin, Denys and Ahmadi, Maryam and Kordi, Fatemeh and Saux, Bertrand and Pastick, Neal and Doucette, Peter and Fleckenstein, Rylie and Luanga, Dalton and Corvin, Alex and Granger, Erwan},
52
+ doi = {https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M},
53
+ month = aug,
54
+ title = {{Prithvi-100M}},
55
+ url = {https://github.com/nasa-impact/Prithvi-100M},
56
+ year = {2023}
57
+ }
58
+ ```