A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation

Robin Camarasa10000-0001-9151-6763, Daniel Bos10000-0001-8979-2603, Jeroen Hendrikse20000-0001-6384-6140, Paul Nederkoorn30000-0001-7386-4545, M. Eline Kooi4, Aad van der Lugt10000-0002-6159-2228, Marleen de Bruijne1,50000-0002-6328-902X
1: Erasmus MC, Rotterdam, 2: University Medical Center Utrecht, 3: Academic Medical Center University of Amsterdam, 4: CARIM School for Cardiovascular Diseases, Maastricht University Medical Center, 5: University of Copenhagen, Denmark
Publication date: 2021/09/28
PDF · Code · Python package · Video · arXiv


Uncertainty assessment has gained rapid interest in medical image analysis. A popular technique to compute epistemic uncertainty is the Monte-Carlo (MC) dropout technique. From a network with MC dropout and a single input, multiple outputs can be sampled. Various methods can be used to obtain epistemic uncertainty maps from those multiple outputs. In the case of multi-class segmentation, the number of methods is even larger as epistemic uncertainty can be computed voxelwise per class or voxelwise per image. This paper highlights a systematic approach to define and quantitatively compare those methods in two different contexts: class-specific epistemic uncertainty maps (one value per image, voxel and class) and combined epistemic uncertainty maps (one value per image and voxel). We applied this quantitative analysis to a multi-class segmentation of the carotid artery lumen and vessel wall, on a multi-center, multi-scanner, multi-sequence dataset of Magnetic Resonance (MR) images. We validated our analysis over 144 sets of hyperparameters of a model. Our main analysis considers the relationship between the order of the voxels sorted according to their epistemic uncertainty values and the misclassification of the prediction. Under this consideration, the comparison of combined uncertainty maps reveals that the multi-class entropy and the multi-class mutual information statistically out-perform the other combined uncertainty maps under study (the averaged entropy, the averaged variance, the similarity Bhattacharya coefficient and the similarity Kullback-Leibler divergence). In a class-specific scenario, the one-versus-all entropy statistically out-performs the class-wise entropy, the class-wise variance and the one versus all mutual information. The class-wise entropy statistically out-performs the other class-specific uncertainty maps in term of calibration. We made a python package available to reproduce our analysis on different data and tasks.


segmentation · carotid artery · uncertainty · bayesian deep learning

Bibtex @article{melba:2021:013:camarasa, title = "A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation", author = "Camarasa, Robin and Bos, Daniel and Hendrikse, Jeroen and Nederkoorn, Paul and Kooi, M. Eline and van der Lugt, Aad and de Bruijne, Marleen", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "UNSURE2020 special issue", year = "2021", pages = "1--39", issn = "2766-905X", doi = "https://doi.org/10.59275/j.melba.2021-ec49", url = "https://melba-journal.org/2021:013" }
RISTY - JOUR AU - Camarasa, Robin AU - Bos, Daniel AU - Hendrikse, Jeroen AU - Nederkoorn, Paul AU - Kooi, M. Eline AU - van der Lugt, Aad AU - de Bruijne, Marleen PY - 2021 TI - A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation T2 - Machine Learning for Biomedical Imaging VL - 1 IS - UNSURE2020 special issue SP - 1 EP - 39 SN - 2766-905X DO - https://doi.org/10.59275/j.melba.2021-ec49 UR - https://melba-journal.org/2021:013 ER -

2021:013 cover