How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model

Hanxue Gu1Orcid, Haoyu Dong1, Jichen Yang1, Maciej A. Mazurowski1,2,3,4
1: Department of Electrical and Computer Engineering Duke University, NC, USA, 2: Department of Radiology Duke University, NC, USA, 3: Department of Computer Science Duke University, NC, USA, 4: Department of Biostatistics & Bioinformatics Duke University, NC, USA
Publication date: 2025/05/08
https://doi.org/10.59275/j.melba.2025-86a6
PDF · Code

Abstract

Automated segmentation is a fundamental medical image analysis task, which enjoys significant advances due to the advent of deep learning. While foundation models have been useful in natural language processing and some vision tasks for some time, the foundation model developed with image segmentation in mind—Segment Anything Model (SAM)—has been developed only recently and has shown similar promise. However, there are still no sys- tematic analyses or “best-practice” guidelines for optimal fine-tuning of SAM for medical image segmentation. This work summarizes existing fine-tuning strategies with various backbone architectures, model components, and finetuning algorithms across 18 combinations and evaluates them on 17 datasets covering all common radiology modalities. Our study reveals that (1) fine-tuning SAM leads to slightly better performance than previous segmentation methods, (2) fine-tuning strategies that use parameterefficient learning in both the encoder and decoder are superior to other strategies, (3) network architecture has a small impact on final performance, and (4) further training SAM with self-supervised learning can improve final model performance. We also demonstrate the ineffectiveness of some methods popular in the literature and further expand our experiments into few-shot and prompt-based settings. Lastly, we released our code and MRI-specific fine-tuned weights, which consistently obtained superior performance over the original SAM, at https://github.com/mazurowskilab/finetune-SAM

Keywords

foundation model · medical image segmentation · Model finetuning · Segment Anything Model

Bibtex @article{melba:2025:006:gu, title = "How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model", author = "Gu, Hanxue and Dong, Haoyu and Yang, Jichen and Mazurowski, Maciej A.", journal = "Machine Learning for Biomedical Imaging", volume = "3", issue = "May 2025 issue", year = "2025", pages = "88--120", issn = "2766-905X", doi = "https://doi.org/10.59275/j.melba.2025-86a6", url = "https://melba-journal.org/2025:006" }
RISTY - JOUR AU - Gu, Hanxue AU - Dong, Haoyu AU - Yang, Jichen AU - Mazurowski, Maciej A. PY - 2025 TI - How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Model T2 - Machine Learning for Biomedical Imaging VL - 3 IS - May 2025 issue SP - 88 EP - 120 SN - 2766-905X DO - https://doi.org/10.59275/j.melba.2025-86a6 UR - https://melba-journal.org/2025:006 ER -

2025:006 cover