Deep spatial-spectral fusion of UAV RGB and hyperspectral imagery for potato plant disease detection

Abstract

Accurate plant disease detection remains challenging when using single-modality data, which fails to capture comprehensive disease-related features. However, many existing studies rely on pixel-level classification or prior plant segmentation and lack explicit modeling of cross-modal interactions, limiting their ability to distinguish between healthy and diseased plants. This study presents a spatial–spectral fusion deep learning framework (S2-PDD) that fuses uncrewed aerial vehicles (UAV)-based RGB imagery and hyperspectral-derived feature representations for plant-level detection of potato diseases (blackleg and Potato virus Y (PVY)), employing early fusion (modalities combined at the input stage) and middle fusion (features integrated at intermediate stages within the model backbone) strategies. The multimodal fusion models were compared against single-modal models and an existing S2ADet model. Model performance, assessed through five-fold cross-validation, demonstrated that multimodal models integrating RGB and vegetation index features achieved the highest mAPs of 86.65 ± 1.71 (%; E-RV model) and 85.74 ± 1.96 (M-RV), respectively. These mAPs were higher than those of all single-modal models, including the RGB-only (83.21 ± 1.46) and hyperspectral-only models (PCA features: 79.71 ± 1.45; vegetation index features: 85.31 ± 2.36). They also exceeded mAPs of multimodal models combining RGB with PCA features (early fusion: 83.00 ± 2.81; middle fusion: 83.11 ± 2.46; S2ADet: 84.04 ± 2.73), regardless of the fusion strategy. The superior performance highlights that vegetation index features provide strong class separability compared to other hyperspectral representations. The proposed models achieved strong plant-level detection performance, with AP of 78.65 ± 4.14 (E-RV) and 77.40 ± 3.46 (M-RV) for blackleg disease, as well as 84.82 ± 3.59 (E-RV) and 83.28 ± 3.86 (M-RV) for PVY. These results demonstrate the potential of UAV-based multimodal sensing for disease monitoring in cropping systems. A potato plant disease detection dataset was constructed and made publicly available, containing paired RGB and hyperspectral image tiles with bounding box annotations. The code is available at https://github.com/Tim-Agro/S2-PDD.

Code and data availability

公開論文であることは確認できましたが、現在の公式API・許可済み取得経路では本文を自動取得できませんでした。

No evidence-backed public reproduction asset is currently recorded.