AutoNUE 2021 Instructions & Benchmark

The challenge will have 5 benchmarks, details of which can be seen below:

  1. Supervised Domain Adaptation
  2. Semi-supervised Domain Adaptation
  3. Weakly supervised Domain Adaptation
  4. Unsupervised Domain Adaptation
  5. Semantic Segmentation

Important Note: We are making the code for Domain Adaptation baselines public here . If anyone is interested, feel free to use it. For Domain Adaptation Challenges, the participants may be requested to submit the code and requirements.txt (containing all the required installations).

I. Supervised Domain Adaptation

This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.

Directions for Participation

  1. Register an account at http://idd.insaan.iiit.ac.in/, with the event selected as "AutoNUE Challenge 2021".
  2. Go to Dataset > Download page in the menu.
  3. Dataset consists of 2 parts which are available for download.
  4. The first part is the IDD Segmentation (IDD 20k Part I).
  5. The other part is IDD Segmentation (IDD 20k Part II).
  6. Extract both the downloaded compressed files into the same folder.
  7. Please run the data preparation code for generating ground truth segmentation masks as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C
  8. Download the target datasets from original websites (all except GTA require registrations), given below for easy reference:
    1. https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
    2. https://bdd-data.berkeley.edu/ (you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
    3. https://download.visinf.tu-darmstadt.de/data/from_games/ (merge all images and labels into a pair folders)
    4. https://www.cityscapes-dataset.com/examples/#fine-annotations (make sure that you only download dataset with fine annotations)
  9. Please run the data preparation code for generating sampled source datasets as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    ./domain_adaptation/source/prep_all.sh

    This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find the images and annotations for the source dataset to be used for this challenge.

  10. Use IDD training and validation datasets for the target.
  11. Once you have built a model, and have the predictions of the model in any of the splits (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size.
  12. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
    1. Sample format is given in the submission link.
    2. The suffix of each file should be "_leftImg8bit".

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code

II. Semi-supervised Domain Adaptation

This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.

Directions for Participation

  1. Register an account at http://idd.insaan.iiit.ac.in/, with the event selected as "AutoNUE Challenge 2021".
  2. Go to Dataset > Download page in the menu.
  3. Dataset consists of 2 parts which are available for download.
  4. The first part is the IDD Segmentation (IDD 20k Part I).
  5. The other part is IDD Segmentation (IDD 20k Part II).
  6. Extract both the downloaded compressed files into the same folder.
  7. Please run the data preparation code for generating ground truth segmentation masks as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --semisup_da True

    Note that only selected train masks, which can be used for this challenge, will be generated for the training stage. All validation masks will be generated for the evaluation stage (refer to step 10 below).

  8. Download the target datasets from original websites (all except GTA require registrations), given below for easy reference:
    1. https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
    2. https://bdd-data.berkeley.edu/ (you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
    3. https://download.visinf.tu-darmstadt.de/data/from_games/ (merge all images and labels into a pair folders)
    4. https://www.cityscapes-dataset.com/examples/#fine-annotations (make sure that you only download dataset with fine annotations)
  9. Please run the data preparation code for generating sampled source datasets as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    ./domain_adaptation/source/prep_all.sh

    This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find the images and annotations for the source dataset to be used for this challenge.

  10. Once you have built a model, and have the predictions of the model in any of the splits (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size.
  11. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
    1. Sample format is given in the submission link.
    2. The suffix of each file should be "_leftImg8bit".

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code

III. Weakly-supervised Domain Adaptation

This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.

Directions for Participation

  1. Register an account at http://idd.insaan.iiit.ac.in/, with the event selected as "AutoNUE Challenge 2021".
  2. Go to Dataset > Download page in the menu.
  3. Dataset consists of 2 parts which are available for download.
  4. The first part is the IDD Segmentation (IDD 20k Part I).
  5. The other part is IDD Segmentation (IDD 20k Part II).
  6. Extract both the downloaded compressed files into the same folder.
  7. Please run the data preparation code for generating ground truth segmentation masks as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --weaksup_da True

    Note that only validation masks will be generated for this challenge towards the evaluation stage (refer to step 11 below). Bounding box annotations to be used for training in this challenge are present here: https://github.com/AutoNUE/public-code/target/weakly-supervised

  8. Download the target datasets from original websites (all except GTA require registrations), given below for easy reference:
    1. https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
    2. https://bdd-data.berkeley.edu/ (you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
    3. https://download.visinf.tu-darmstadt.de/data/from_games/ (merge all images and labels into a pair folders)
    4. https://www.cityscapes-dataset.com/examples/#fine-annotations (make sure that you only download dataset with fine annotations)
  9. Please run the data preparation code for generating sampled source datasets as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    ./domain_adaptation/source/prep_all.sh

    This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find the images and annotations for the source dataset to be used for this challenge.

  10. Once you have built a model, and have the predictions of the model in any of the splits (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size.
  11. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
    1. Sample format is given in the submission link.
    2. The suffix of each file should be "_leftImg8bit".

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code

IV. Unsupervised Domain Adaptation

This challenge involves domain adaptation from around 20k samples of Mapillary, Cityscapes (fine annotations only), Berkeley Deep Drive, and GTA as the source dataset (S) to the IDD as target dataset (T). For the IDD dataset, participants have to submit the results for Level-3 (26 classes) hierarchy.

Directions for Participation

  1. Register an account at http://idd.insaan.iiit.ac.in/, with the event selected as "AutoNUE Challenge 2021".
  2. Go to Dataset > Download page in the menu.
  3. Dataset consists of 2 parts which are available for download.
  4. The first part is the IDD Segmentation (IDD 20k Part I).
  5. The other part is IDD Segmentation (IDD 20k Part II).
  6. Extract both the downloaded compressed files into the same folder.
  7. Please run the data preparation code for generating ground truth segmentation masks as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C --unsup_da True

    Note that only validation masks will be generated for this challenge towards the evaluation stage (refer to step 11 below). IDD Training labels cannot be used for this challenge. Images from training data can be used.

  8. Download the target datasets from original websites (all except GTA require registrations), given below for easy reference:
    1. https://www.mapillary.com/dataset/vistas?pKey=q0GhQpk20wJm1ba1mfwJmw
    2. https://bdd-data.berkeley.edu/ (you might have to click on Advanced tab, and then click on "proceed to bdd-data.berkeley.edu")
    3. https://download.visinf.tu-darmstadt.de/data/from_games/ (merge all images and labels into a pair folders)
    4. https://www.cityscapes-dataset.com/examples/#fine-annotations (make sure that you only download dataset with fine annotations)
  9. Please run the data preparation code for generating sampled source datasets as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    ./domain_adaptation/source/prep_all.sh

    This will create the folder public-code/domain_adaptation/source/source_datasets_dir/source_datasets_dir/ where you will find the images and annotations for the source dataset to be used for this challenge.

  10. Once you have built a model, and have the predictions of the model in any of the splits (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size.
  11. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
    1. Sample format is given in the submission link.
    2. The suffix of each file should be "_leftImg8bit".

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code

V. Semantic Segmentation

The segmentation challenge involves pixel level predictions for all the 26 classes at level 3 of the label hierarchy (see Overview, for details of the level 3 ids).

Directions for Participation

  1. Register an account at http://idd.insaan.iiit.ac.in/, with the event selected as "AutoNUE Challenge 2021".
  2. Go to Dataset > Download page in the menu.
  3. Dataset consists of 2 parts which are available for download.
  4. The first part is the IDD Segmentation (IDD 20k Part I).
  5. The other part is IDD Segmentation (IDD 20k Part II).
  6. Extract both the downloaded compressed files into the same folder.
  7. Please run the data preparation code for generating ground truth segmentation masks as documented here: https://github.com/AutoNUE/public-code. Use the following command for segmentation mask generation:
    python preperation/createLabels.py --datadir $ANUE --id-type level3Id --num-workers $C
  8. Once you have built a model, and have the predictions of the model in any of the splits (train, val), you can evaluate the metric as directed here: https://github.com/AutoNUE/public-code#evaluation. Use the following command for segmentation evaluation:
    python evaluate/evaluate_mIoU.py --gts $GT --preds $PRED --num-workers $C
    Your prediction is a png image, which has the size of 1280x720. Each pixel of this image contains the label as level 3ds (see labels code) of the corresponding image (resized to 1280x720). The evaluation code above resizes both your prediction and ground truth png files to 1280x720, in case they are not of that size.
  9. Finally you can upload the predictions for the test split (4k; 2k each from the two parts of IDD20K), to be evaluated for the leaderboard here: http://idd.insaan.iiit.ac.in/evaluation/submission/submit/
    1. Sample format is given in the submission link.
    2. The suffix of each file should be "_leftImg8bit".

Output Format

The output format is a png image with the same resolution as the input image, where the value of every pixel is an integer in {0. .... , 26}, where the first 0-25 classes correspond to the level 3 ids (see Overview, for details of the level 3 ids) and the class 26 is used as a miscellaneous class.

Metric

We will be using the mean Intersection over Union metric. All the ground truth and predictions maps will be resized to 720p (using nearest neighbor) and True positives (TP), False Negatives (FN) and False positives (FP) will be computed for each class (except 26) over the entire test split of the dataset. Intersection over Union (IoU) will be computed for each class by the formula TP/(TP+FN+FP) and the mean value is taken as the metric (commonly known as mIoU) for the segmentation challenge.

Additionally we will also be reporting the mIoU for level 2 and level 1 ids also at 720p resolution in the leaderboard. Evaluation scripts are available here: https://github.com/AutoNUE/public-code