Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey

Jérôme Rony10000-0002-6359-6142, Soufiane Belharbi10000-0001-6326-380X, Jose Dolz20000-0002-2436-7750, Ismail Ben Ayed1, Luke McCaffrey3, Eric Granger10000-0001-6116-7945
1: LIVIA, Dept. of Systems Engineering, École de technologie supérieure, Montreal, Canada, 2: LIVIA, Dept. of Software and IT Engineering, École de technologie supérieure, Montreal, Canada, 3: Goodman Cancer Research Centre, Dept. of Oncology, McGill University, Montreal, Canada
Publication date: 2023/03/06
PDF · Code · arXiv


Using state-of-the-art deep learning (DL) models to diagnose cancer from histology data presents several challenges related to the nature and availability of labeled histology images, including image size, stain variations, and label ambiguity. In addition, cancer grading and the localization of regions of interest (ROIs) in such images normally rely on both image- and pixel-level labels, with the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of DL models. Given only image-class annotations, these methods can be trained to simultaneously classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of deep WSOL methods to identify and locate diseases in histology images, without the need for pixel-level annotations. We propose a taxonomy in which these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen only limited progress, recent bottom-up methods are currently driving a lot of progress with the use of deep WSOL methods. Early works focused on designing different spatial pooling functions. However, those methods quickly peaked in term of localization accuracy and revealed a major limitation, namely, – the under-activation of CAMs, which leads to high false negative localization. Subsequent works aimed to alleviate this shortcoming and recover the complete object from the background, using different techniques such as perturbation, self-attention, shallow features, pseudo-annotation, and task decoupling.
In the present paper, representative deep WSOL methods from our taxonomy are also evaluated and compared in terms of classification and localization accuracy using two challenging public histology datasets – one for colon cancer (GlaS), and a second, for breast cancer (CAMELYON16). Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges posed by histology data often use priors such as ROI size, or additional pixel-wise supervision estimated from a pre-trained classifier, allowing them to achieve better results. However, all the methods suffer from high false positive/negative localization. Classification performance is mainly affected by the model selection process, which uses either the classification or the localization metric. Finally, four key challenges are identified in the application of deep WSOL methods in histology, namely, – under-/over-activation of CAMs, sensitivity to thresholding, and model selection – and research avenues are provided to mitigate them. Our code is publicly available at


Medical/Histology Image Analysis · Computer-Aided Diagnosis · Deep Learning · Weakly Supervised Object Localization · Weakly Supervised Learning · Image Classification

Bibtex @article{melba:2023:004:rony, title = "Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey", author = "Rony, Jérôme and Belharbi, Soufiane and Dolz, Jose and Ben Ayed, Ismail and McCaffrey, Luke and Granger, Eric", journal = "Machine Learning for Biomedical Imaging", volume = "2", issue = "March 2023 issue", year = "2023", pages = "96--150", issn = "2766-905X", doi = "", url = "" }
RISTY - JOUR AU - Rony, Jérôme AU - Belharbi, Soufiane AU - Dolz, Jose AU - Ben Ayed, Ismail AU - McCaffrey, Luke AU - Granger, Eric PY - 2023 TI - Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey T2 - Machine Learning for Biomedical Imaging VL - 2 IS - March 2023 issue SP - 96 EP - 150 SN - 2766-905X DO - UR - ER -

2023:004 cover