AgriFusionNet: a context-aware multimodal leaf disease diagnosis and classification system for sustainable plant health monitoring

Abstract

Early and accurate identification of plant diseases is essential for improving crop productivity and ensuring food security. Many existing deep learning-based plant disease classification methods rely solely on leaf images collected from a controlled environment, which limits their applicability in real-world agricultural conditions where symptoms may be visually unclear and influenced by environmental factors. To address these challenges, this study discusses AgriFusionNet, a context-aware multimodal deep learning framework that integrates leaf images, textual symptom descriptions, and environmental data for robust plant disease classification. The proposed architecture employs EfficientNet-B0 for visual feature extraction, BERT for semantic representation of symptom descriptions, and a lightweight multilayer perceptron for modeling environmental factors such as temperature, humidity, rainfall, and soil moisture. Features from all three modalities are fused into a unified representation to train the CNN model. The model is trained and tested upon the Context-Aware Multimodal Augmented PlantVillage dataset covering 38 plant diseases and healthy classes. Experimental results show that AgriFusionNet gives an overall accuracy of 98.94% on the dataset Context-Aware Multimodal Augmented PlantVillage, with competitive precision and recall and F1-score. The multimodal framework facilitates the co-learning of visual, semantic, and contextual environmental representations and the analyses of the confusion matrix and feature interactions give insights into cross-modal relationships. The proposed approach aims to explore context-aware multimodal representation learning for agricultural AI applications, with emphasis on integrating complementary visual, semantic, and contextual information.

Code and data availability

The paper's data availability statement points to the Context-Aware Multimodal Augmented PlantVillage dataset (leaf images, symptom text, environmental data used for the phenotyping/classification analysis) deposited publicly on IEEE Dataport with a DOI matching an allowed URL.