Question about the validity of the truth masks
Hello,
We are wondering about the accuracy of the truth masks in a number of images in the training and evaluation folds.
For instance, there appears to be a large burned area on the bottom left corner of this image (fold 4, image 27), yet the truth mask ignores it,
There is also image (fold 0, image 9) which -to our understanding- shows a large burned area, though the mask doesn't show it.
Finally the most puzzling is this one
Is this perhaps because the burned land is not recent enough, or perhaps an incomplete label, or is it certain that the masks are all correct and we are misjudging these areas?
Thanks
The answer to your observations can be:
- The burned area was too old
- A cause different from a wildfire burned the area
The images refer to a wildfire with a containment date at most one month before.
I have investigated 497f80b4-878a-4f97-9522-5edcd5229dfb_1 (fold 0, image 9). It is what we can consider a limit case; the big burned area you are mentioning is related to an ongoing wildfire that will be contained two months later.
So far, we have built a model that detects burned land, but there was no mention about containment dates and that we should only detect areas that were declared as contained in a recent timespan. As you mention, one month. There is no conclusive way to identify containment dates from sentinel bands, as this is a human decision. We added a way to detect active fires on burned areas but then we notice there is a mask that detects an area where the smoke is still visible as positive.
So here are two seperate images showing the same region with a different truth, both images show two burned regions.
The first truth mask indicates that the burned area on the left has recently been contained, while the burned area on the right is still uncontained.
The second truth mask indicates that the right area has recently been contained, while the area on the left is now too old to be labeled. But the area on the right is at least as old as the one on the left. Both show no signs of smoke or active fire.
Before we work further on a solution please provide us some better understanding about the exact challenge in this competition.
Thank you
Thanks for your notes.
The task is identifying areas burned by a wildfire. In these cases, we do not have access to a day-by-day annotation. We cannot assume the burned area of an ongoing wildfire is the same as the damaged area of the completely extinguished wildfire (which is the information we have).
In this last case, the images are from two dates with a distance of two months. These are rare cases when two wildfires start one near the other at two different times, which will be contained (and extinguished) at different times.
We can describe the use case in this way: you know more or less where a wildfire happened, and you want to understand the affected areas to plan the restoration process. Today these annotations are manually made by someone, and you want a system to do this work with a satellite. Mapping the inner area of a large wildfire is quite challenging using on-site annotation.
We trained the baseline with the same issues to make a fair comparison.