Cellpose Stringer et al. (2021) has now become a standard in cell segmentation. Its excellent perfomance, processing speed, and ergonomic graphical interface make it a handy tool for every day cell biology image analysis. However, it occasionally fails in scenarios involving complex and elongated objects. In such cases, it tends to produce over-segmentation, where neighboring objects are split in smaller fragments.

The Omnipose algorithm Cutler et al. (2022) was conceived in order to address this limitation. The main difference between Omnipose and Cellpose is the fact that the distance map is defined as the distance to the cell boundaries in Omnipose, while it is defined as a distance to a cell “centroid” in Cellpose. A weakness of the latter is that there is no canonical choice to define this center, hence Omnipose’s choice seems more principled. This explains our decision to choose and base our work on its architecture.

Figure 2 summarizes the main ideas behind the Omnipose architecture and its training. Omnipose is based on a regular convolutional neural network (CNN), with a U-Net like architecture Ronneberger et al. (2015). Given an input 2D image with $N$ pixels, the CNN can be seen as a mapping $N_{w}^{\textrm{omni}}$ of the form

It depends on weights $w$ that should be optimized during a training stage. It returns $3$ different outputs (illustrated on the top of Figure 2):

• •

  $N_{w}^{b}(u)\equiv$ boundary probability: at every pixel, the value of this image can be interpreted as a probability of being a boundary between the objects to segment.

• •

  $N_{w}^{d}(u)\equiv$ distance map: at a given pixel, the value of this map is equal to:

  • –

    The distance of the pixel to the closest object boundary, if the pixel is inside an object.

  • –

    $0$ (or a fixed negative value) elsewhere.

• •

  $N_{w}^{\mathbf{v}}(u)\equiv$ flow field: can be interpreted as the gradient of the distance map. It is an essential feature of the Cellpose and Omnipose architectures. Ultimately, the flow is used through a procedure called Euler integration to generate a segmentation mask. This is illustrated on the top right of Figure 2.

The original training stage involves a collection of $K\in\mathbb{N}$ images $(u_{k})_{1\leq k\leq K}$ together with their exhaustive segmentation masks. For every image $u_{k}$ in the dataset, an algorithm creates the gold standard boundary probability $b_{k}^{\star}$, distance map $d_{k}^{\star}$ and flow field $\mathbf{v}_{k}^{\star}$. This is illustrated on the bottom of Figure 2.

The weights $w$ of the neural network are then optimized so as to minimize a loss function that compares the output of the CNN with the gold standard:

where $b_{k}=N_{w}^{b}(u_{k})$, $d_{k}=N_{w}^{d}(u_{k})$, $\mathbf{v}_{k}=N_{w}^{\mathbf{v}}(u_{k})$.

In the original Omnipose implementation available on GitHub, the different losses were defined as follows:

• •

  Boundary loss $\ell_{\mathcal{B}}$:

  This term compares the predictions $b$ to $b^{\star}$ using the following loss:

  $$\ell_{\mathcal{B}}(b,b^{\star})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\frac{\lambda_{\mathcal{B}}}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}g(b[\mathbf{x}],b^{\star}[\mathbf{x}]),$$

  (4)

  where $g:\mathbb{R}\times\mathbb{R}\to\mathbb{R}$ combines a sigmoid and a binary cross entropy loss.

• •

  Distance loss $\ell_{\mathcal{D}}$:

  This loss calculates a weighted mean squared error between the predicted distance fields and the ground truth distance fields. It is defined as

  $$\ell_{\mathcal{D}}(d,d^{\star})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\frac{\lambda_{\mathcal{D}}}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}(d[\mathbf{x}]-d^{\star}[\mathbf{x}])^{2}\cdot\rho[\mathbf{x}],$$

  where $\rho\in\mathbb{R}^{N}$ is a weight image with higher values around the gold standard boundaries.

• •

  Flow loss $\ell_{\mathcal{V}}$:

  This loss is defined as a weighted sum of three losses $\ell_{\mathcal{V}}=\ell_{\mathcal{V}}^{1}+\ell_{\mathcal{V}}^{2}+\ell_{\mathcal{V}}^{3}$. The first one is a mean squared error loss:

  $$\ell_{\mathcal{V}}^{1}(\mathbf{v},\mathbf{v}^{\star})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\frac{\lambda_{\mathcal{V},1}}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}\left\|\mathbf{v}[\mathbf{x}]-\mathbf{v}^{\star}[\mathbf{x}]\right\|_{2}^{2}\cdot\rho[\mathbf{x}].$$

  (5)

  The second one compares the norms of the vector fields:

  $$\ell_{\mathcal{V}}^{2}(\mathbf{v},\mathbf{v}^{\star})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\frac{\lambda_{\mathcal{V},2}}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}(\|\mathbf{v}[\mathbf{x}]\|_{2}-\|\mathbf{v}^{\star}[\mathbf{x}]\|)^{2}\cdot\rho[\mathbf{x}].$$

  (6)

  The third one aims to minimize the distance between trajectories generated through the ground truth and predicted flows. Trajectories starting from an initial point $\mathbf{z}$ can be generated by simple explicit Euler discretization:

  	$\displaystyle\mathbf{x}_{0}(\mathbf{z})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\mathbf{z}$
  	$\displaystyle\mathbf{x}_{l+1}(\mathbf{z})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\mathbf{x}_{l}(\mathbf{z})+\Delta t\cdot\mathbf{v}[\mathbf{x}_{l}(\mathbf{z})]$

  	$\displaystyle\mathbf{x}^{\star}_{0}(\mathbf{z})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\mathbf{z}$
  	$\displaystyle\mathbf{x}^{\star}_{l+1}(\mathbf{z})$	$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\mathbf{x}_{l}^{\star}(\mathbf{z})+\Delta t\cdot\mathbf{v}^{\star}[\mathbf{x}^{\star}_{l}(\mathbf{z})].$

  where $\Delta t$ is a step-size. Letting $L\in\mathbb{N}$ denote an integration time, the “Euler” loss then becomes:

  $$\ell_{\mathcal{V}}^{3}(\mathbf{v},\mathbf{v}^{\star})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\frac{\lambda_{\mathcal{V},3}}{|\mathcal{X}|}\sum_{\mathbf{z}\in\mathcal{X}}\sum_{l=1}^{L}\|\mathbf{x}_{l}(\mathbf{z})-\mathbf{x}_{l}^{\star}(\mathbf{z})\|_{2}^{2}.$$

  (7)

  It measures how two trajectories generated by Euler integration using the ground truth and predicted vector fields deviate. This loss is implemented in the torchVF library by Peters (2022). For more information, we refer the reader to the related report.

An inspection of the code reveals that the different weights have been set empirically as: $\lambda_{\mathcal{B}}=10$, $\lambda_{\mathcal{D}}=2$, $\lambda_{\mathcal{V},1}=2$, $\lambda_{\mathcal{V},2}=2$, $\lambda_{\mathcal{V},3}=1$.

The notations are summarized in Table 1. We assume that the domain $\mathcal{X}=\mathcal{X}_{0}\sqcup\mathcal{X}_{1}$ is partitioned with the background set $\mathcal{X}_{0}$ and the foreground set $\mathcal{X}_{1}$. A difficulty in instance segmentation is that multiple objects may exist within the connected components of a region $\mathcal{X}_{i}$. To differentiate them, we let $(\mathcal{X}_{i,j})_{1\leq j\leq J_{i}}$ denote a partition of the set $\mathcal{X}_{i}$ as different objects within a similar class. For instance in Figure 3(a), the foreground set $\mathcal{X}_{1}$ is split in 13 components. A connected component of $\mathcal{X}_{1}$ can be split as $\mathcal{X}_{1,2}\cup\mathcal{X}_{1,3}$. The background set $\mathcal{X}_{0}$ is split in a single component $\mathcal{X}_{0,1}$.

We let

denote the set of all edges (or object boundaries) within the image. It is depicted in red in Figure 3(a).

The input of our neural network is a set of “sketches” or strokes drawn by the user. We let $\mathcal{S}_{0}$ and $\mathcal{S}_{1}$ denote the strokes describing the background and foreground respectively. They are depicted in brown and blue respectively in Figure 3(b). The intersection of the brown and blue strokes define natural boundaries. We can indeed construct the touching boundaries $\mathcal{B}_{0,1}$ between different strokes as

where $\overline{\mathcal{X}}$ is the closure of $\mathcal{X}$ in the continuous setting and the interface between neighboring pixels in the discrete setting.

In addition, the user can delineate other boundaries, denoted $\mathcal{B}_{\textrm{manual}}$, to separate touching objects within a class. We can concatenate all the boundaries to obtain a complete boundary set $\mathcal{B}$ defined as

For the algorithm to work properly, we require the following set of assumptions.

The main result we will use to define and certify our algorithm is summarized in the following theorem.

The proof of this theorem is given in Appendix A. This theorem should be understood as follows. The first identity informs us that we can compute the exact distance map $\mathrm{dist}(x,\mathcal{E})$ to the set of exact boundaries $\mathcal{E}$ on the valid set $\mathcal{D}$. This set can be computed using only the partial annotations of the boundaries $\mathcal{B}\subseteq\mathcal{E}$ and the different semantic regions $\mathcal{S}_{i}\subseteq\mathcal{X}_{i}$. The second inequality tells us that we have an information everywhere on the strokes $\mathcal{S}_{0}$ and $\mathcal{S}_{1}$. Moreover, in the case of total annotations, we get $\mathcal{D}=\mathcal{X}$ and the proposed idea will lead to a training equivalent to the one in Omnipose and we can see the proposed setting as a generalization. Figure 4 schematically summarizes Theorem 4.

For this experiment, the model have been trained for 100 epochs ($\approx$ 2 minutes) with a batch size of 16 and image flips for data augmentation.

In the example of Figure 5, we use a microscopy image of methicillin-resistant Staphylococcus aureus (MRSA) infections, from European Commission, Horizon Magazine (2020), “Can we reverse antibiotic resistance?”. It is reused under the European Commission’s reuse policy.

After drawing for less than one minute and training for 100 epochs ($\approx$ 2’), we achieve a much better result than the trained model of Omnipose (see Figure 5). The quality metrics is shown in Figure 5(d).

In this section, we picked an image from the Omnipose dataset, which likely represents eggs of an insect on a tree leaf. At first sight, the segmentation task is uneasy, since the objects are tightly connected, with identical textures and blurry boundaries. We first annotated a subset of 5 eggs in Figure 6(b) with a minimal amount of background. The segmentation result after training is already surprisingly good in Figure 6(d), but some objects are not detected, and others are merged. We annotated 2 extra eggs in Figure 6(c). With this extra information, retraining the network now produces a near perfect segmentation mask, with a single error (2 pink eggs on the left). This experiment illustrates a unique feature of Sketchpose: it is possible to interactively annotate while training. This offers a possibility to label a minimum amount of regions to reach the desired output. This principle sometimes called “active learning” or “human-in-the-loop” Budd et al. (2021) is significantly enhanced by using partial annotations and the user-friendly Napari interface.

As for the previous experiment, the model have been trained for 100 epochs ($\approx$ 2 minutes) with a batch size of 16 and image flips for data augmentation.

Bacteria are often used as biological models (e.g. in DNA studies). A precise segmentation can be difficult to achieve, because they have elongated shapes and can be clustered.

The Omnipose Cutler et al. (2022) model was initially conceived to address the shortcomings of Cellpose for this task.

Figure 7 shows how transfer learning with sparse annotations can improve the Omnipose results by separating touching bacterias. Figure 7(d) shows a quantitative comparison of both methods. As can be seen, Sketchpose’s adapted weights provide much higher performance. A visual inspection indicates that all objects have been correctly separated, apart from the cluster touching the boundary on the bottom left.

The image in Figure 7(e) shows a crop of a very large image of a skin explant provided by DIVA Expertise. One can see a part of the dermis (in pink) and above it, adipose tissue (large white circular cells). Adipose tissue is the third skin layer after the epidermis and dermis, also known as the hypodermis. Hypodermal cells (adipocytes) secrete specific molecules (e.g. adiponectin, leptin) which have a direct impact on the biology of fibroblasts present in the dermis, and also on keratinocytes present in the epidermis. They are the subject of numerous studies (see Bourdens et al. (2019) and Sadick et al. (2015) for instance). For most of the studies where skin explants are imaged, we first need to count the adipocytes number in the image, and remove any potential outliers detected in the dermis and epidermis.

While Omnipose cyto2 results in some undersegmentation for this task, the adapted weights provided by Sketchpose yields significantly enhanced results. Annotating 6 cells and a training for 100 epochs (less than 1 minute) were sufficient to significantly improve the quality of the segmentation and to remove the outliers from the dermis (see Figure 7(c)). Figure 7 shows a quantitative comparison between Omnipose and Sketchpose on this example.

Osteoclasts are responsible for bone resorption, and are widely studied (see Labour et al. (2016) for instance) as being responsible for certain pathologies such as osteoporosis when dysfunctional. Their differentiation goes through several stages, culminating in the activated osteoclast. The latter is generally large and contains numerous nuclei. Atlantic Bone Screen (ABS) company is investigating the effect of different drugs in inducing either proliferation or cell death in these activated osteoclasts, in order to regulate their population. To do so, they extract osteoclasts from biopsies, culture them, apply the drugs and image them under a bright-field microscope.

The studied image is a crop of an image containing around 20,000 cells. We can see touching cells presenting a great variety in size, shape color. The image is complex to segment and poses a real challenge. What is more, ABS does not want to count pre-osteoclasts (small black nuclei), but only the mature cells (according to specific nuclei criteria). Each study comprises around sixty images, hence manual counting task performed at ABS is costly and laborious.

In Figure 7, we present a qualitative depiction that underscores the enhancement in segmentation accuracy attained through transfer learning with just a few labels. Labeling required approximately 2 minutes, while the training process took about 5 minutes. A quantitative comparison is available in Figure 7(l).

In this section, we investigate the model robustness across various annotation levels each characterized by a different percentage of annotated pixels: 10%, 25%, 50%, and 100%. We generate randomly binary masks by thresholding white Gaussian noise with a Gaussian filter of variance $\sigma^{2}$. The resulting Gaussian process is then thresholded to keep only a given proportion of pixels.

While the model is stochastic in nature, the generated data is created once and for all, enabling its deterministic reuse across multiple training sessions. Figure 9 shows an image accompanied by four corresponding label masks illustrating decreasing levels of annotation sparsity.

For each dataset, we trained the Sketchpose model for 1000 epochs. This takes about 5 hours for the Microbeseg dataset and 30 hours for the PanNuke dataset using our Nvidia RTX5000. Each model was trained with the four percentages of annotated pixels we described above. There was no data augmentation, except random cropping of the images to 224x224 pixels. This is the size which was used in the original Omnipose model.

We evaluated and compared the performance using two alternative models. The first one is the Cellpose 3.0 model Stringer and Pachitariu (2025), which regresses a distance to the objects centroids. The second one is the LKCell model Cui et al. (2024), which is a better performing variant of CellVit Hörst et al. (2024), itself a variant of HoverNet Graham et al. (2019). These models are among the most popular and best performing for histopathology images such as the PanNuke dataset. While they perform classification and segmentation, we will just compare their ability to segment objects, as we are not interested in the classification task here. We trained both LKCell and Cellpose 3.0 from scratch on each of the two datasets with complete annotations for 1000 epochs.

We compare the performance using other standard quality metrics used in instance segmentation. In all the metrics below, an IoU threshold of 50% is used.

• •

  Precision: measures how many of the predicted positives are actually correct (i.e., the fraction of predicted segments that are true).

• •

  Recall: how many of the actual positives were correctly predicted (i.e., how complete the prediction is).

• •

  F1-Score: harmonic mean of precision and recall, balancing both.

• •

  Detection Quality (DQ): evaluates object-level detection performance, penalizing missed or extra objects. It evaluates the ability to detect object instances correctly, regardless of segmentation quality.

• •

  Segmentation Quality (SQ): measures how well matched objects are segmented, assuming correct pairing, reflecting the quality of the predicted segment.

The main results are reported in Table 2, and several key observations emerge.