QCell: Recombining and Aligning Cell Queries
for Overlapping Instance Segmentation

BMVC 2026

Yaroslav Prytula1,2, Anton Popov2,3, Dmytro Fishman1,4,5
1Institute of Computer Science, University of Tartu, Tartu, Estonia
2Faculty of Applied Sciences, Ukrainian Catholic University, Lviv, Ukraine
3Department of Electronic Engineering, Micro- and Biomedical Electronics,
Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine
4STACC OÜ, Tartu, Estonia
5Better Medicine OÜ, Tartu, Estonia
Paper HF Dataset Code Poster Slides Video
QCell architecture overview

Overview of QCell. QCell builds on a MaskDINO-style query-based segmentation pipeline, where multi-scale image features and positional embeddings are processed by the encoder and refined by transformer decoder layers with content and DN queries. The proposed modules are shown above: (a) instance recombination decomposes each query into amodal, visible, and occluded sub-representations and recombines them into a refined full-instance query; (b) contrastive query learning uses matched instance queries q̂i by Hungarian matching as anchors, corresponding DN queries d̂+i across all groups as positives and other DN queries as negatives to align queries of the same cell and separate queries of different cells in latent space.

Abstract

Instance segmentation of overlapping cells in microscopy remains challenging due to semi-transparent structures that produce weak boundaries and mixed visual evidence in overlap regions. Existing methods address this through local regions of interest or shape priors but lack global reasoning across overlapping objects. We present QCell, a novel query-based model that de-overlaps cell instances in microscopy scenes. Our approach combines (i) an instance recombination module that decomposes and recombines query representations in latent space, enabling the model to reason about complete object structure under overlap, and (ii) a contrastive query alignment objective that combines distinctive instance feature learning and separation of overlapping cell queries. We additionally introduce a new Organoid dataset benchmark for overlapping cell segmentation. We show that QCell outperforms state-of-the-art methods across multiple benchmarks, achieving +2.2 AP and +2.7 AJI on ISBI2014.

Organoids images and ground-truth instance annotations

Organoids. Example images and ground-truth annotations from the organoids dataset.

Organoids. One of our key contributions is a novel Organoids dataset for overlapping object segmentation in brightfield microscopy. The dataset contains 1,186 training images, 1,199 validation images, and 201 test images at a resolution of 540 × 540. The dataset presents dense and highly overlapping scenes, with up to 105 instances per training image and an average of 96 instances per test image, reaching a maximum of 223. This makes Organoids a challenging real-world benchmark for evaluating instance separation and de-overlapping in microscopy. Additional dataset details are provided in the supplementary material.

BibTeX

<TODO>

Acknowledgements

The authors acknowledge the support of the European Union and the Estonian Research Council through project TEM-TA101. Computational resources were provided by the High-Performance Computing Cluster at the University of Tartu 🇪🇪. We thank the Biomedical Computer Vision Lab for its invaluable support. We thank Revvity and the Institut de Recherche en Santé Digestive (IRSD), Inserm UMR 1220, Toulouse, France, for jointly providing the Organoids dataset and supporting its annotation and validation. We express our gratitude to the Armed Forces of Ukraine 🇺🇦 and the bravery of the Ukrainian people for enabling a secure working environment, without which this work would not have been possible.