A residual forecasting framework for plant dynamic growth based on cross-modal spatial alignment

Abstract

Plant phenotyping is essential for modern crop breeding, yet traditional static image analysis fails to capture the nonlinear dynamics of plant growth. Existing time-series forecasting models exhibit notable limitations when processing multimodal data: global pooling operations may compress local 2D spatial topology of plants, and shallow feature concatenation may be insufficient for effective cross-modal semantic alignment. Moreover, current methods typically regress absolute morphological states, which may contribute to temporal lag during nonlinear growth spurts. In this paper, we propose ST-CrossGro-Former, a cross-modal residual forecasting framework for plant dynamic growth. The network removes the final global pooling and classification layers to preserve spatial topology and incorporates a scalar-guided cross-modal attention module based on the standard query-key-value formulation. This module utilizes 1D morphological features as queries to dynamically weight local visual regions, promoting multimodal feature alignment. Concurrently, a residual incremental forecasting strategy is introduced to predict short-term growth increments rather than absolute states, aiming to improve tracking sensitivity to sudden growth events. Evaluations on the UNL-CPPD maize dataset and supplementary validation on the FIP1 wheat field dataset show that the proposed model achieves competitive single-step forecasting accuracy and favorable temporal trajectory alignment compared with adapted spatiotemporal attention, graph-based, and physics-informed baselines under the evaluated settings. In particular, the FIP1 results suggest that ST-CrossGro-Former can maintain favorable height trajectory alignment under a field-acquired wheat setting, indicating its potential for helping mitigate temporal misalignment in dynamic growth forecasting.

Code and data availability

The paper evaluates its ST-CrossGro-Former model on two public plant phenotyping datasets: the UNL-CPPD maize dataset (explicitly stated as publicly available with a repository URL) and the FIP1 wheat field dataset (public dataset from ETH Zürich, with its GigaScience dataset publication DOI). No author analysis code,