Journal of Cardiology Research and Endovascular Therapy
Voice and Gesture-Based Interfaces for Telemedicine Platforms: An ML-Driven HCI Approach for Rural Healthcare Accessibility
Abstract
Idowu Olugbenga Adewumi, Hameed Qudus Alabi and Sarah Adanini
This research introduces a multimodal telehealth platform created for rural and underprivileged populations, incorporating automatic speech recognition (ASR) and gesture detection. A collection of 16,500 samples (9,200 speech, 7,300 gesture) was gathered from 3 rural dialects and 6 gestures of clinical significance. The ASR system reached a word error rate (WER) of ≤ 10% (clean: 7.0%, noisy: 9.0%), and the gesture classifier garnered an overall accuracy of 92.3%, with class-specific F1-scores between 90.1% and 95.8%. Latency tests conducted on low-end Android devices (2 GB RAM, quad-core CPU) indicated an average response time of 276 ms (SD = 14 ms). A usability study (N = 52 participants, ages 21–63 years, 54% female) indicated a System Usability Scale (SUS) score of 82.6 (SD = 6.3), surpassing the standard “excellent” usability benchmark (> 80). In three case studies of rural clinics, the platform decreased consultation duration by 23–31% and enhanced patient–provider understanding scores by 18–25%. The diagnostic visualization featured a confusion matrix (6 × 6) exhibiting 92% diagonal dominance, a ROC curve with AUC = 0.957, and an error analysis revealing only a +2% WER decline in noisy environments. These findings indicate that the suggested system is adaptable, quick, and robust, showcasing significant potential to enhance global health equity, in line with SDG 3 (Health), SDG 4 (Education), and SDG 10 (Inequalities). In general, the system shows that multimodal interaction is not only technically possible but also socially significant. Its ability to scale in low-resource settings and rural areas makes it a viable solution for enhancing global health equity, closely aligning with SDG 3 (Health), SDG 4 (Education), and SDG 10 (Reduced Inequalities).