Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization

Franz Thaler10000-0002-6589-6560, Christian Payer20000-0002-5558-9495, Martin Urschler30000-0001-5792-3971, Darko Štern10000-0003-3449-5497
1: Medical University of Graz, Austria, 2: Graz University of Technology, Austria, 3: The University of Auckland, New Zealand
Publication date: 2021/09/24
PDF · Code · Video · arXiv


In landmark localization, due to ambiguities in defining their exact position, landmark annotations may suffer from large observer variabilities, which result in uncertain annotations. To model the annotation ambiguities of the training dataset, we propose to learn anisotropic Gaussian parameters modeling the shape of the target heatmap during optimization. Furthermore, our method models the prediction uncertainty of individual samples by fitting anisotropic Gaussian functions to the predicted heatmaps during inference. Besides state-of-the-art results, our experiments on datasets of hand radiographs and lateral cephalograms also show that Gaussian functions are correlated with both localization accuracy and observer variability. As a final experiment, we show the importance of integrating the uncertainty into decision making by measuring the influence of the predicted location uncertainty on the classification of anatomical abnormalities in lateral cephalograms.


inter-observer variability · fully-convolutional neural network · heatmap regression · uncertainty estimation · landmark localization

Bibtex @article{melba:2021:014:thaler, title = "Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization", author = "Thaler, Franz and Payer, Christian and Urschler, Martin and Štern, Darko", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "UNSURE2020 special issue", year = "2021", pages = "1--27", issn = "2766-905X", doi = "", url = "" }
RISTY - JOUR AU - Thaler, Franz AU - Payer, Christian AU - Urschler, Martin AU - Štern, Darko PY - 2021 TI - Modeling Annotation Uncertainty with Gaussian Heatmaps in Landmark Localization T2 - Machine Learning for Biomedical Imaging VL - 1 IS - UNSURE2020 special issue SP - 1 EP - 27 SN - 2766-905X DO - UR - ER -

2021:014 cover