The BraTS-PEDs 2023 dataset was supported by multiple consortia, including Children’s Brain Tumor Network (CBTN, https://cbtn.org/) (Lilly et al., 2023), International DIPG/DMG Registry (https://www.dipgregistry.org) and COllaborative Network for NEuro-oncology Clinical Trials (CONNECT, https://connectconsortium.org/), along with additional clinical centers, including Boston’s Children Hospital and Yale University. All cases contained pre- and post-gadolinium T1-weighted (labeled as T1 and T1CE, respectively), T2-weighted (T2), and T2-weighted fluid attenuated inversion recovery (FLAIR) images. These conventional mpMRI sequences are commonly acquired as part of standard clinical care for brain tumors. However, the image acquisition protocols, and MRI equipment differ across different institutions, resulting in heterogeneity in image quality and appearance in the multi-consortium cohort. Inclusion criteria comprised of pediatric subjects with: (1) histologically approved high-grade glioma, i.e., astrocytoma and DMG, including radiologically or histologically proven DIPG; (2) availability of all four structural mpMRI sequences on treatment-naive imaging sessions. Exclusion criteria included: (1) images that were of low quality or contained artifacts precluding reliable tumor segmentation, as determined by qualitative control (QC) by researchers experienced in pediatric brain tumor imaging and segmentation; and (2) infants younger than one month of age. The latter criteria was imposed as the neonatal brain within the first month undergoes rapid developmental changes, including immature myelination and evolving tissue contrasts (Gilmore et al., 2007), which can complicate image segmentation and analysis. In the 2023 BraTS-PEDs, data for 167 patients were included (CBTN, n = 113; Boston’s Children Hospital, n = 30; Yale University, n = 24). Data from the International DIPG/DMB Registry was only processed but not segmented for the 2023 challenge, but will be included in the 2024 challenge.

For all patients, mpMRI scans were pre-processed using a standardized approach, including conversion of the DICOM files to the NIfTI file format, co-registration to the same anatomical template (i.e., SRI24) (Rohlfing et al., 2010), and resampling to an isotropic resolution of 1 mm3. The pre-processing pipeline is publicly available through the Cancer Imaging Phenomics Toolkit (CaPTk) (Pati et al., 2020; Rathore et al., 2018; Davatzikos et al., 2018), Federated Tumor Segmentation (FeTS) tool https://github.com/FETS-AI/Front-End/ (Pati et al., 2022), and the docker container for pediatric-specific pipeline including skull-stripping and tumor segmentation on GitHub: https://github.com/d3b-center/peds-brain-seg-pipeline-public. De-identification was performed through removing protected health information from DICOM headers. Defacing was performed via skull-stripping using a pediatric-specific skull-stripping method (Fathi Kazerooni et al., 2023a) to prevent any potential facial reconstruction/recognition of the patients. Figure 1A presents the data preparation workflow.

The pre-processed images were segmented into tumor subregions using either of two pediatric automated deep learning segmentation models (Fathi Kazerooni et al., 2023a; Liu et al., 2023) before being manually corrected by pediatric neuro-radiologists. The tumors were segmented into four subregions. These subregions comprising of enhancing tumor (ET), non-enhancing tumor (NET), cystic component (CC), and peritumoral edema (ED) regions, with definitions provided in (Familiar et al., 2024), and summarized below:

• •

  ET is described by areas with enhancement (brightness) on T1 post-contrast images as compared to T1 pre-contrast. In case of mild enhancement, checking the signal intensity of normal brain structures can be helpful.

• •

  CC typically appears with hyperintense signal (very bright) on T2 and hypointense signal (dark) on T1CE. The cystic portion should be within the tumor, either centrally or peripherally (as compared to ED which is peritumoral). The brightness of CC is defined as comparable or close to cerebrospinal fluid (CSF).

• •

  NET is defined as any other abnormal signal intensity within the tumorous region that cannot be defined as enhancing or cystic. For example, the abnormal signal intensity on T1, FLAIR, and T2 that is not enhancing on T1CE should be considered as non-enhancing portion.

• •

  ED is defined by the abnormal hyperintense signal (very bright) on FLAIR scans. ED is finger-like spreading that preserves underlying brain structure and surrounds the tumor.

The American Society of Neuroradiology collaborated in generating ground truth annotation for the majority of data in this challenge (ASNR, https://www.asnr.org/). Four labels were generated (Figure 1) using the preliminary automated segmentation and were then manually revised by ASNR volunteer neuroradiology experts of varying rank and experience in accordance with the annotation guidelines. The expert annotators were provided with the four mpMRI sequences (T1, T1CE, T2, FLAIR) along with the fused automated segmentation volume to initiate the manual refinements using the ITK-SNAP software (Yushkevich et al., 2006). After segmentation corrections by the annotators, three attending board-certified neuroradiologists reviewed the segmentations to either approve or return to the individual annotators for further refinements. This process was followed iteratively until the approvers found the refined tumor subregion segmentations acceptable for public release and the challenge conduction. The final segmentations were provided and used as ground truth for model training and evaluation.

Although four tumor subregions were annotated for all the pediatric data, to follow the guidelines of the other BraTS 2023 challenges and to allow the teams to potentially incorporate data from adult glioma challenge into their model training, the teams were provided with only three segmentation labels, i.e., non-enhancing component (NC – the union of non-enhancing tumor, cystic component, and necrosis; label 1), ED (label 2), and ET (label 3), as shown in Figure 1B. The color maps in the images shown in Figure 1B illustrate the label descriptions provided in the original annotations given to the annotators (Figure 1B; left), as compared to the final annotations (Figure 1B; right) – which adhere to the standards of other BraTS challenges – provided to the teams.

The BraTS-PEDs 2023 cohort was split into training (n = 98), validation (n = 44), and testing datasets (n = 24). Training and validation included a random split between CBTN and Boston cases. All Yale cases were used for independent testing. The training data shared with the teams comprised mpMRI scans and ground truth labels for the training cohort, released in mid-May 2023. The cohort shared for validation included only the mpMRI sequences without associated ground truth and it was released in late June 2023. Notably, the testing cohort was withheld from the participants.

Teams were prohibited from training their algorithm on any additional public and/or private data besides the provided BraTS-PEDs 2023 data or to using models pretrained on other datasets. This restriction was imposed to allow for a direct and fair comparison among the participating methods. However, teams were allowed to use additional public and/or private data for publication of their scientific papers, on the condition that they also provide a report of their results on the data from BraTS-PEDs 2023 challenge alone and discuss potential differences in the obtained results.

The methods submitted to the BraTS-PEDs 2023 challenge were evaluated based on performances for the segmentation of ET, tumor core (TC – the combination of ET and NC), and the entire tumorous region, or whole tumor (WT – the union of ET, NC, and ED) (Figure 1C). The containerized submissions of the teams were evaluated on the synapse.org platform supported by Medperf (Karargyris et al., 2023). Teams were required to send the output of their methods to the evaluation platform for the scoring to occur during the training and the validation phases. At the end of the validation phase, teams were asked to identify the method they would like to be evaluated by the organizers in the final testing/ranking phase. The training, validation, and testing phases ran between May-July 2023, July-August 2023, and August 2023, respectively. The results and ranking of the challenge were presented at the MICCAI 2023 conference on October 10, 2023.

The evaluation and ranking were based on a lesionwise ranking scheme in which the ranking for each team was computed relative to other challenge competitors for each of the subjects in the test set, for each tumor region, e.g., ET, TC, and WT. We used two established performance evaluation metrics, i.e., Dice similarity coefficient (Dice) and 95% Hausdorff distance (HD95) using the Generally Nuanced Deep Learning Framework (GaNDLF; gandlf.org) (Pati et al., 2023). At BraTS-PEDs 2023, each team was ranked for 24 test subjects, 3 tumor regions, and 2 metrics, resulting in 144 individual rankings. Then for each team, the final ranking score (FRS) was computed by producing a cumulative rank as an average of all individual rankings for each patient, followed by averaging the cumulative ranks across all testing subjects for each competing team.

The lesion-wise Dice and HD95 were calculated by isolating and identifying each disjoint lesion, through dilating the ground truth segmentation masks for each tumor region, i.e., ET, TC, and WT, by 3 pixels in all directions (i.e., by a size of 3x3x3). Dice and HD95 scores were calculated for each component or lesion individually, after penalizing all false positive and false negative values with a score of 0. Subsequently, the mean value of Dice and HD95 scores across all lesions were calculated for each subject. The lesions below a cutoff value of 50 voxels were removed from any further evaluation. If the models were not able to predict a region for a subject, the Dice score was assigned a value of 0 and HD95 was set to 374 (the distance between corners in the SRI atlas space), and if they correctly did not predict a non-existing region, the Dice score and HD95 were given values of 1 and 0, respectively. Lesion-wise Dice and HD95 scores for each case were calculated according to equations 1 and 2. More information on the evaluation metrics along with the codes can be found at https://github.com/rachitsaluja/BraTS-2023-Metrics.

Where L represents the number of ground truth lesions, TP denotes the number of true positives, FN indicates the number of false negatives, and FP signifies the number of false positives.

Finally, the statistical differences in performance, beyond those expected by chance, between each pair of teams were determined using permutation testing. This was achieved for each pair of teams, by randomly permuting the cumulative ranks for each subject and calculating the difference in FRS between the pair of teams in each permutation. Statistical significance of rankings was determined using a p-value when the proportion of times the difference in FRS using the permutated data exceeded the observed difference in FRS using the actual data. These values are reported in an upper triangular matrix.

The training data, including ground truth segmentations, and the validation data without ground truth segmentations can be accessed and downloaded from the Synapse www.synapse.org/Synapse:syn51156910/wiki/627000. More information about the BraTS-PEDs 2023 challenge is at www.synapse.org/Synapse:syn51156910/wiki/622461.

Among the teams that participated in the training phase, 18 teams submitted a total of 159 entries during the validation phase, with 147 of these submissions receiving scores. Nine teams completed all challenge stages and were evaluated during the testing phase of the BraTS‐PEDs 2023 challenge (Capellán-Martín et al., 2023; Myronenko et al., 2023; Zhou et al., 2023; Huang et al., 2023; Maani et al., 2023; Mistry et al., 2023; Javaji et al., 2023; Mantha et al., 2023; Temtam et al., 2023). During the competition, participants were allowed unlimited validation of their algorithms on the Synapse platform using validation data with withheld ground truth, which enabled them to iteratively fine-tune their models before making their final submission for testing. The validation outcomes for each team is reported in their respective publications (Capellán-Martín et al., 2023; Myronenko et al., 2023; Zhou et al., 2023; Huang et al., 2023; Maani et al., 2023; Mistry et al., 2023; Javaji et al., 2023; Mantha et al., 2023; Temtam et al., 2023).

The participating teams in the BraTS-PEDs 2023 challenge utilized diverse deep learning-based segmentation approaches, incorporating various architectures, training strategies, and post-processing techniques to tackle the complexities of pediatric brain tumor segmentation. Below, we provide a summarized overview of these methods (see Table 1 for a summary of the proposed techniques by each team), with further details available in the individual publications (Capellán-Martín et al., 2023; Myronenko et al., 2023; Zhou et al., 2023; Huang et al., 2023; Maani et al., 2023; Mistry et al., 2023; Javaji et al., 2023; Mantha et al., 2023; Temtam et al., 2023).

Several teams used variants of the U-Net architecture, with modifications to improve feature extraction and segmentation accuracy. The BiomedMBZ team implemented SegResNet and MedNext with deep supervision and extensive pre- and post-processing (Maani et al., 2023). Blackbean introduced the Scalable and Transferable U-Net (STU-Net), a large-scale model with up to 1.4 billion parameters, pre-trained on the TotalSegmentator dataset (Huang et al., 2023).

Several teams adopted ensemble approaches to improve robustness. CNMCPMMI combined nnU-Net with Swin UNETR, applying label-wise ensembling and cross-validated thresholding to refine segmentation outputs (Capellán-Martín et al., 2023). SchlaugLab developed a model combining ONet and modified U-Net variants, integrating custom loss functions and composite data augmentation (Javaji et al., 2023). SherlockZyb extended nnU-Net with self-supervised pre-training and an adaptive region-specific loss function to handle tumor heterogeneity and class imbalances (Zhou et al., 2023).

Automated model selection was explored by NVAUTO, which utilized Auto3DSeg from MONAI, requiring minimal user input while training SegResNet with 5-fold cross-validation and deep supervision (Myronenko et al., 2023). Meanwhile, UMNIverse introduced a GAN-based model, Temporal Cubic PatchGAN (TCuP-GAN), incorporating Convolutional Long Short-Term Memory Networks (ConvLSTMs) to leverage temporal MRI features (Mantha et al., 2023). Other teams focused on novel architectural modifications. Isahajmistry team enhanced nnU-Net with Omni-Dimensional Dynamic Convolution (ODConv3D) and multi-scale attention mechanisms for improved feature learning (Mistry et al., 2023). VisionLab proposed a Multiresolution Fractal Deep Neural Network (MFDNN), using wavelet-based convolution layers to extract tumor textures at multiple scales while performing uncertainty analysis to enhance segmentation reliability (Temtam et al., 2023).

Table 1 provides a summary of the cumulative ranks and FRS for each team across all test subjects, including the teams’ rankings, and a high-level description of the methods used by each team. The CNMCPMI2023 team (Capellán-Martín et al., 2023) achieved the highest score among the teams, followed by NVAUTO (Myronenko et al., 2023), SherlockZyb (Zhou et al., 2023), and Blackbean (Huang et al., 2023) (see Table 1 that illustrates the scores and methods used by all teams that completed the challenge tasks).

Figure 2 illustrates the p-values calculated between pairs of teams to evaluate statistical significance between the achieved performance. Specifically, among the best performing teams, the results for CNMCPMI2023 and NVAUTO (p = 0.45), NVAUTO and SherlockZyb (p = 0.1), and SherlockZyb and Blackbean (p = 0.43) were not significantly different. This may be due to the small sample size of the test cohort (n = 24) for this challenge.

Figures 3 to 5 display the Dice and HD95 scores for all test subjects for all the participating teams. Tables 2 to 4 summarize the performance metrics—Dice, HD95, and sensitivity—of each team. The CNMCPMI2023 $|$ NVAUTO $|$ SherlockZyb $|$ Blackbean teams achieved Dice scores of 0.83 $|$ 0.84 $|$ 0.83 $|$ 0.81 for WT, 0.81 $|$ 0.78 $|$ 0.77 $|$ 0.79 for TC, and 0.65 $|$ 0.55 $|$ 0.63 $|$ 0.53 for ET segmentation, and HD95 values of 20.86 $|$ 18.05 $|$ 6.11 $|$ 23.56 for WT, 21.82 $|$ 27.10 $|$ 22.28 $|$ 22.02 for TC, and 43.89 $|$ 115.32 $|$ 59.68 $|$ 22.02 for ET segmentation. Interestingly, the sensitivity of the NVAUTO algorithm for the segmentation of WT and ET was higher than that of the other two teams, indicating a higher overall rate of true positive predictions by this algorithm. However, this increased sensitivity came at the cost of over-segmentation, resulting in more false positives and less accurate boundary placement for ET segmentation.

The prediction results for ET demonstrate a greater variability and skewness in the Dice score distribution, as indicated by the mean and median values in Table 2 and the box plots in Figure 3. In contrast, the segmentation of TC and WT (Figures 4-5) shows less variability and skewness. Given the small volume or absence of the ET region in DMG tumors, the Dice score more heavily penalizes segmentation errors in the ET region. TC and WT segmentations are more consistent and accurate across all teams, with WT segmentation being the most precise. This indicates that, on average, most methods reliably locate the lesion within MRI scans, while distinguishing between different tumor compartments is more challenging.

Figure 6 showcases examples of the performances of CNMCPMI2023, NVAUTO, SherlockZyb, and Blackbean teams for tumor segmentation in four testing subjects. Panels (A-B) indicate the best results for these three teams, with WT Dice above 0.90. Panels (C-D) present the subjects with worst results across the top performing teams. As indicated in these examples, there are situations where one algorithm performs better than the remaining algorithms in segmentation of tumorous regions. In Figure 6(C), CNMCPMI2023, SherlockZyb, and Blackbean achieved Dice scores of 0.88, while NVAUTO achieved a low Dice score (0.18) for WT segmentation. This low performance for NVAUTO was achieved due to many false positive results for segmentation of the tumor for this subject. On the other hand, in the example presented in Figure 6(D), NVAUTO and Blackbean algorithms produced WT segmentation results with Dice scores of 0.60 and 0.70, respectively, while the other two algorithms, i.e., CNMCPMI2023 and SherlockZyb, did not generate any segmentation outputs (WT Dice = 0 or 0.05). This situation underscores the potential benefit of ensembling different algorithms in achieving better overall performance.

In 2022, the BraTS Glioma challenge tested top-performing methods—developed for adult glioma segmentation—on a small set of unseen pediatric brain tumor data. Results revealed that although models trained on adult brain tumors achieved high Dice scores for segmenting the whole pediatric tumor from mpMRI scans, their performance on tumor subregions (such as ET or NC) was inaccurate. This confirmed our hypothesis that effective pediatric segmentation techniques require training on a curated and annotated pediatric brain tumor dataset. However, as with any machine learning and deep learning endeavor, developing a model that generalizes well necessitates a large, standardized dataset. To address this, the BraTS-PEDs 2023 initiative has compiled the most extensive annotated collection of pediatric high-grade gliomas, including astrocytomas and DMGs.

The top-performing methods at BraTS-PEDs 2023 included an ensemble method combining nnU-Net and Swin UNETR, Auto3DSeg using the SegResNet algorithm, and an extension of the nnU-Net model with self-supervised pretraining integrated with adaptive region-specific loss. All of these methods are based on the U-Net convolutional neural network architecture. The models achieved mean Dice scores of 0.83-0.84 for WT segmentation and slightly lower yet respectable Dice scores of 0.77-0.81 for TC segmentation. However, lower and more variable performance was observed in the segmentation of the ET region, with mean Dice scores of 0.65, 0.63 and 0.55 across the three top performing teams, which emphasizes the unique challenges posed by pediatric brain tumors. The small volume or absence of ET region in DMG likely contributes to the greater variability and skewness in the Dice score distribution for this subregion. This finding indicates that while most methods can reliably locate the WT within MRI scans, accurately distinguishing between different tumor compartments remains a significant challenge in pediatric brain tumors.

The examples in the manuscript demonstrate that no single algorithm consistently outperforms others across all cases. The variability in performance, especially in challenging scenarios, highlights the need for continued research and development in this field. Specifically, ensembling the top-performing models may achieve better overall performance for segmentation of multi-institutional data by leveraging their individual strengths and mitigating their weaknesses. For instance, a method with high sensitivity, despite relatively lower Dice and higher HD95 scores, can be useful when it is critical not to miss any positive regions (such as in ET segmentation). By combining this method with others, it can be refined to reduce false positives and improve boundary accuracy.

The advancements in segmentation algorithms, as showcased by the BraTS-PEDs 2023 challenge, have several clinical implications. Detailed automated segmentations can enable radiation oncologists to better target treatment areas while sparing healthy tissue, ultimately reducing side effects (Bibault and Giraud, 2024). Furthermore, the integration of such automated tools into clinical workflows may expedite diagnostic processes and reduce inter-observer variability, facilitating more standardized and timely treatment decisions. Importantly, consistent monitoring facilitated by these algorithms could lead to prompt interventions, which is particularly crucial in pediatric populations where timely treatment is essential to reduce long-term adverse developmental impacts. Similarly, accurate tumor delineation can improve surgical planning by providing neurosurgeons with precise maps of tumor boundaries, thereby improving resection strategies and minimizing damage to critical brain structures (Sulangi et al., 2024). Future work should focus on prospective clinical validations to assess the real-world benefits and cost-effectiveness, ensuring they contribute meaningfully to patient outcomes and personalized treatment strategies.

Unlike previous challenges (e.g., BRATS (Baid et al., 2021; Moawad et al., 2023; LaBella et al., 2024b)) that focused on adult populations, the BraTS‐PEDs 2023 challenge specifically addressed brain tumor segmentation in pediatric patients, who present unique challenges such as age-related anatomical variations. Nonetheless, our analysis reaffirms common limitations of current deep learning paradigms, including limited generalizability across diverse acquisition protocols and dependence on large, well-annotated datasets, that continue to impede robust clinical translation.

BraTS-PEDs 2023 also faced specific limitations, including a small dataset size and a narrow range of pediatric brain tumor histologies. For instance, the statistical analysis comparing the top teams revealed no significant performance differences, which may be attributed to the small sample size of the test cohort (n = 24). Future challenges would benefit from larger test cohorts to better discern performance differences and validate algorithm robustness.

In addition, to align with the rest of the BraTS 2023 cluster of challenges, our evaluation focused on segmenting the enhancing tumor, tumor core, and whole tumor. This focus may limit applicability of these methods for response assessment in pediatric brain tumors. Recognizing the distinct differences between adult and pediatric brain tumors (Mackay et al., 2017), especially in components critical for response assessment (Mackay et al., 2017; Thakkar et al., 2014; Familiar et al., 2024), we will customize future evaluations to better address pediatric tumor characteristics. Furthermore, the typically small size of ET regions in DMG tumors prompted us to set a threshold of 50 voxels for the detection of a true positive, which might have impacted the accuracy of ET predictions. Moving forward, we intend to adopt a more data-driven approach to setting this threshold in subsequent challenges.

Encouraged by the results of the BraTS-PEDs 2023 challenge, our next steps include expanding the dataset to include subjects from additional institutions and incorporating various histologies of high- and low-grade pediatric glioma. This initiative will offer the research community access to a comprehensive dataset of rare pediatric tumors with curated mpMRI and annotation, thereby aiding in the development of advanced tools for computer-assisted treatment planning and prognosis. Future data collection will also encompass post-operative and post-treatment scans and clinical outcomes. The data is publicly available at www.synapse.org/Synapse:syn51156910/wiki/627000.