Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs

Sina Amirrajab1Orcid, Yasmina Al Khalil1, Cristian Lorenz2, Jürgen Weese2, Josien Pluim1, Marcel Breeuwer1,3
1: Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands, 2: Philips Research Laboratories, Hamburg, Germany, 3: Philips Healthcare, MR R&D - Clinical Science, Best, The Netherlands
Publication date: 2023/06/08
https://doi.org/10.59275/j.melba.2023-1g8b
PDF · Code · arXiv

Abstract

We propose a method for synthesizing cardiac magnetic resonance (MR) images with plausible heart pathologies and realistic appearances for the purpose of generating labeled data for the application of supervised deep-learning (DL) training. The image synthesis consists of label deformation and label-to-image translation tasks. The former is achieved via latent space interpolation in a VAE model, while the latter is accomplished via a label-conditional GAN model. We devise three approaches for label manipulation in the latent space of the trained VAE model; i) intra-subject synthesis aiming to interpolate the intermediate slices of a subject to increase the through-plane resolution, ii) inter-subject synthesis aiming to interpolate the geometry and appearance of intermediate images between two dissimilar subjects acquired with different scanner vendors, and iii) pathology synthesis aiming to synthesize a series of pseudo-pathological synthetic subjects with characteristics of a desired heart disease. Furthermore, we propose to model the relationship between 2D slices in the latent space of the VAE prior to reconstruction for generating 3D-consistent subjects from stacking up 2D slice-by-slice generations. We demonstrate that such an approach could provide a solution to diversify and enrich an available database of cardiac MR images and to pave the way for the development of generalizable DL-based image analysis algorithms. We quantitatively evaluate the quality of the synthesized data in an augmentation scenario to achieve generalization and robustness to multi-vendor and multi-disease data for image segmentation. Our code is available at https://github.com/sinaamirrajab/CardiacPathologySynthesis

Keywords

Cardiac Pathology Synthesis · Image Synthesis · Conditional GANs · VAEs

Bibtex @article{melba:2023:010:amirrajab, title = "Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs", author = "Amirrajab, Sina and Al Khalil, Yasmina and Lorenz, Cristian and Weese, Jürgen and Pluim, Josien and Breeuwer, Marcel", journal = "Machine Learning for Biomedical Imaging", volume = "2", issue = "June 2023 issue", year = "2023", pages = "288--311", issn = "2766-905X", doi = "https://doi.org/10.59275/j.melba.2023-1g8b", url = "https://melba-journal.org/2023:010" }
RISTY - JOUR AU - Amirrajab, Sina AU - Al Khalil, Yasmina AU - Lorenz, Cristian AU - Weese, Jürgen AU - Pluim, Josien AU - Breeuwer, Marcel PY - 2023 TI - Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs T2 - Machine Learning for Biomedical Imaging VL - 2 IS - June 2023 issue SP - 288 EP - 311 SN - 2766-905X DO - https://doi.org/10.59275/j.melba.2023-1g8b UR - https://melba-journal.org/2023:010 ER -

2023:010 cover