Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data

Ayaan Haque1, Abdullah-Al-Zubaer Imran20000-0001-5215-339X, Adam Wang2, Demetri Terzopoulos3,4
1: Saratoga High School, Saratog, 2: Stanford University, Stanford, 3: University of California, Los Angeles, 4: VoxelCloud, Inc., Los Angeles
Publication date: 2021/10/27
PDF · Code · arXiv


Deep learning-based models, when trained in a fully-supervised manner, can be effective in performing complex image analysis tasks, although contingent upon the availability of large labeled datasets. Especially in the medical imaging domain, however, expert image annotation is expensive, time-consuming, and prone to variability. Semi-supervised learning from limited quantities of labeled data has shown promise as an alternative. Maximizing knowledge gains from copious unlabeled data benefits semi-supervised learning models. Moreover, learning multiple tasks within the same model further improves its generalizability. We propose MultiMix, a new multi-task learning model that jointly learns disease classification and anatomical segmentation in a semi-supervised manner while preserving explainability through a novel saliency bridge between the two tasks. Our experiments with varying quantities of multi-source labeled data in the training sets confirm the effectiveness of MultiMix in the simultaneous classification of pneumonia and segmentation of the lungs in chest X-ray images. Moreover, both in-domain and cross-domain evaluations across these tasks further showcase the potential of our model to adapt to challenging generalization scenarios. Our code is available at


Multi-Task Learning · Semi-Supervised Learning · Data Augmentation · Saliency Bridge · Classification · Segmentation · Chest X-Ray · Lungs · Pneumonia

Bibtex @article{melba:2021:011:haque, title = "Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data", author = "Haque, Ayaan and Imran, Abdullah-Al-Zubaer and Wang, Adam and Terzopoulos, Demetri", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "October 2021 issue", year = "2021", pages = "1--25", issn = "2766-905X", doi = "", url = "" }
RISTY - JOUR AU - Haque, Ayaan AU - Imran, Abdullah-Al-Zubaer AU - Wang, Adam AU - Terzopoulos, Demetri PY - 2021 TI - Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data T2 - Machine Learning for Biomedical Imaging VL - 1 IS - October 2021 issue SP - 1 EP - 25 SN - 2766-905X DO - UR - ER -

2021:011 cover