Abstract:Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments. Traditional vision-based approaches face significant challenges regarding power consumption, computational latency, and user privacy. This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors. We used an 8 times 8 multizone ToF sensor (VL53L8CH) and an 8 times 8 IR array (AMG8833) to capture complementary depth and thermal cues. A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller (MCU). Experimental results on a custom dataset of 7 static gestures, validated via k-fold cross-validation, demonstrate that the proposed fusion strategy significantly outperforms single-sensor baselines with an accuracy of 92.3% and a macro F1-score of 0.93. Finally, on-device benchmarks on STM32F4 and STM32H7 MCUs confirm the system's suitability for resource-constrained wearables, requiring only 6,343 parameters and achieving millisecond-level inference latency with a total system power of 50 mW.
| Comments: | The article is already accepted for IEEE Sensors Applications Symposium (IEEE SAS) 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.13462 [cs.LG] |
| (or arXiv:2605.13462v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13462 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Christian Veronesi [view email]
[v1]
Wed, 13 May 2026 12:53:22 UTC (5,807 KB)