Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data

Ayaan HaqueSaratoga High School, Saratog, Abdullah-Al-Zubaer ImranStanford University, Stanford, Adam WangStanford University, Stanford, Demetri TerzopoulosUniversity of California, Los Angeles
VoxelCloud, Inc., Los Angeles
October 2021 issue
Publication date: 2021/10/27
PDF · arXiv · Code


Deep learning-based models, when trained in a fully-supervised manner, can be effective in performing complex image analysis tasks, although contingent upon the availability of large labeled datasets. Especially in the medical imaging domain, however, expert image annotation is expensive, time-consuming, and prone to variability. Semi-supervised learning from limited quantities of labeled data has shown promise as an alternative. Maximizing knowledge gains from copious unlabeled data benefits semi-supervised learning models. Moreover, learning multiple tasks within the same model further improves its generalizability. We propose MultiMix, a new multi-task learning model that jointly learns disease classification and anatomical segmentation in a semi-supervised manner while preserving explainability through a novel saliency bridge between the two tasks. Our experiments with varying quantities of multi-source labeled data in the training sets confirm the effectiveness of MultiMix in the simultaneous classification of pneumonia and segmentation of the lungs in chest X-ray images. Moreover, both in-domain and cross-domain evaluations across these tasks further showcase the potential of our model to adapt to challenging generalization scenarios. Our code is available at


Multi-Task Learning · Semi-Supervised Learning · Data Augmentation · Saliency Bridge · Classification · Segmentation · Chest X-Ray · Lungs · Pneumonia

Bibtex @article{melba:2021:011:haque, title = "Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data", authors = "Haque, Ayaan and Imran, Abdullah-Al-Zubaer and Wang, Adam and Terzopoulos, Demetri", journal = "Machine Learning for Biomedical Imaging", volume = "1", issue = "October 2021 issue", year = "2021" }

2021:011 cover