Ans: The anonymization process is described in detail in section 3.4 of the Sewer-ML paper. Shortly summarized: We fine-tuned a Faster-RCNN on ~23k images, where all text boxes are annotated. We then apply that to all 1.3 million images, and apply a heavy Gaussian blur to all detected text.
For the VA label, you said that VA is annotated at the start and end of the inspection video as well as when the water level changes within a 10% step interval. Could you let me know why you've decided to group them? And what indicators you used to suggest that the water level has changed within a 10% step interval?
Ans: All labels in the Sewer-ML dataset follows the Danish Sewer inspection guide called the “Fotomanual”, specifically the 6th edition. In this standard the water level is annotated as described in the Sewer-ML paper. However, in the 7th edition of the standard they changed this so that instead the water level is clustered into 4 levels: 0-5%, 5-15%, 15-30%, and 30-100%.
In some follow up papers we have tried comparing these two labeling approaches, and found the 4 levels led to better results:
Also, for a short intro to the Danish inspection standard, I would recommend reading Section 2 of my PhD Dissertation
In your paper, you mentioned two Danish standards (2010 and 2015). In the 2010 standard, the water is annotated in 10% intervals. However, in your dataset, there are water levels of 1%, 2%, 3%, 4%, and 5%. I was wondering how you can get the water level to be precise as 1%. (waiting for the reply)
Ans: This is an artifact of the human subjectivity in the inspection process, leading to what you could call overly specific classification. When reading the inspection standard the levels are specified as e.g. <5%, between 5% and 15% etc. Therefore, what we do is simply assign the level it is closest to. So for 1-5% that would be 0%. This should also be represented in our released code.
Are you using the same dataset as the Defect classification paper to train, validate, and test your water level, estimation model?
Ans: Yes,we use the same split as the full Sewer-ML dataset
I would like to establish the baseline of my defect classification and water estimation models by applying them to your test dataset. Is it possible to provide me with the test dataset (labels and images) you used to calculate the metric scores of your defect classification and water level estimation models? I have tried to submit my defect classification model's results to the defect classification challenge. However, it seems like the submission port is broken.
Ans: Im very happy to hear your interest in our dataset, and that you are using it. As you know, we currently have problems with our benchmark. Im not in a position where I can share the test ground truth annotations, but the images should be available online. If you need results for a specific model on the test set, you can send me the prediction file (formatted as described on the benchmark website) and I will calculate the metrics. However, we do not want you to use the test set as a second validation set, and therefore request you only submit predictions from your final model. Also, please not that since im currently attending ECCV in Tel Aviv, this will not be possible until next week.