FLiTrak3D: Improved deep-learning-based 3D insect flight kinematics tracking using spatial and temporal encoding

Abstract

Quantitative measurements of insect flight behaviour are essential for understanding the biomechanics, control, and ecology of flight, yet obtaining such measurements under free-flight conditions remains challenging. Small body size, rapid wing motion, visual symmetry, and frequent occlusions complicate three-dimensional pose estimation, often requiring restrictive experimental setups or substantial manual annotation. We present FLiTrak3D, an open-source Python package for estimating insect flight kinematics from multi-view videography. It combines machine-learning-based markerless tracking with biomechanical modelling to reconstruct and parametrise insect body and wing motion. The workflow integrates image preprocessing, including dynamic image cropping and enhancement, two-dimensional bodypart localisation, three-dimensional reconstruction, and optimisation-based skeletal fitting. A key innovation is the use of spatio-temporal encoding across synchronised camera views and adjacent frames to improve neural-network awareness of spatial and temporal context during bodypart localisation. Multi-view image stitching allows the network to use cross-view spatial relationships, reducing left-right bodypart misidentifications, while temporal encoding stacks consecutive greyscale frames into RGB images provide short-term motion information. A species-specific skeleton is then fitted to reconstructed keypoints to enforce kinematic constraints, and estimate body and wing orientations. We demonstrate the approach using free-flying Aedes aegypti mosquitoes, achieving bodypart localisation errors close to human labelling, and realistic flight kinematics.

Code and data availability

植物フェノタイピング解析を再現する公開資産であることを、入力本文と直接リンクから確認できなかったため保留しました。

No evidence-backed public reproduction asset is currently recorded.