Dynamic sparse point voxel transformer for 3D point cloud instance segmentation of dormant apple trees.

Abstract

Three dimensional (3D) instance segmentation is essential for precision characterization of tree architecture at the branch level, which supports both tree fruit crop breeding and the development of robotic systems for orchard management. Existing methods usually use sparse convolution-based operation, which requires a coordinate quantization preprocess to generate sparse tensors, risking the loss of geometric details for fine-grained downstream phenotyping tasks. To overcome this challenge, we developed the dynamic sparse point-voxel transformer (DSPVFormer) model for the efficient and accurate 3D instance segmentation of high-resolution point clouds for dormant apple trees. Our hybrid DSPVFormer architecture maximizes the use of raw point features by dynamically mapping and aggregating the raw point features into the sparse voxel embeddings, capturing strong geometric features that may be discarded during quantization. Evaluations demonstrate that DSPVFormer achieved statistically significant improvements over baseline models on most instance segmentation metrics, which are further translated into more accurate phenotyping evaluation including branch counting and pruning map generation. These advances directly benefit downstream applications in plant phenotyping and robotic pruning for tree crops such as apples. Meanwhile, experimental results on phenotyping tasks suggested that phenotyping-specific evaluation metrics should be prioritized over upstream computer vision performance metrics to realize the full potential of high-throughput phenotyping for real-world applications.

Code and data availability

The paper states that its data and code (including the DSPVFormer analysis pipeline built on Plant Segmentation Studio) are publicly available at the authors' PSS GitHub repository. The COS dataset of 98 dormant apple tree point clouds is also referenced as accessible via this repository/statement. Other URLs (spconv,m