Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs

Sina Amirrajab10000-0001-8226-7777, Yasmina Al Khalil1, Cristian Lorenz2, Jürgen Weese2, Josien Pluim1, Marcel Breeuwer1,3
1: Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands, 2: Philips Research Laboratories, Hamburg, Germany, 3: Philips Healthcare, MR R&D - Clinical Science, Best, The Netherlands
Publication date: 2023/06/08
PDF · Code · arXiv


We propose a method for synthesizing cardiac magnetic resonance (MR) images with plausible heart pathologies and realistic appearances for the purpose of generating labeled data for the application of supervised deep-learning (DL) training. The image synthesis consists of label deformation and label-to-image translation tasks. The former is achieved via latent space interpolation in a VAE model, while the latter is accomplished via a label-conditional GAN model. We devise three approaches for label manipulation in the latent space of the trained VAE model; i) intra-subject synthesis aiming to interpolate the intermediate slices of a subject to increase the through-plane resolution, ii) inter-subject synthesis aiming to interpolate the geometry and appearance of intermediate images between two dissimilar subjects acquired with different scanner vendors, and iii) pathology synthesis aiming to synthesize a series of pseudo-pathological synthetic subjects with characteristics of a desired heart disease. Furthermore, we propose to model the relationship between 2D slices in the latent space of the VAE prior to reconstruction for generating 3D-consistent subjects from stacking up 2D slice-by-slice generations. We demonstrate that such an approach could provide a solution to diversify and enrich an available database of cardiac MR images and to pave the way for the development of generalizable DL-based image analysis algorithms. We quantitatively evaluate the quality of the synthesized data in an augmentation scenario to achieve generalization and robustness to multi-vendor and multi-disease data for image segmentation. Our code is available at


Cardiac Pathology Synthesis · Image Synthesis · Conditional GANs · VAEs

Bibtex @article{melba:2023:010:amirrajab, title = "Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs", author = "Amirrajab, Sina and Al Khalil, Yasmina and Lorenz, Cristian and Weese, Jürgen and Pluim, Josien and Breeuwer, Marcel", journal = "Machine Learning for Biomedical Imaging", volume = "2", issue = "June 2023 issue", year = "2023", pages = "288--311", issn = "2766-905X", doi = "", url = "" }
RISTY - JOUR AU - Amirrajab, Sina AU - Al Khalil, Yasmina AU - Lorenz, Cristian AU - Weese, Jürgen AU - Pluim, Josien AU - Breeuwer, Marcel PY - 2023 TI - Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs T2 - Machine Learning for Biomedical Imaging VL - 2 IS - June 2023 issue SP - 288 EP - 311 SN - 2766-905X DO - UR - ER -

2023:010 cover