AI-Driven Sign Language Recognition System with NLP-Enhanced Transcription

Sign language is a critical communication medium for deaf and hard-of-hearing individuals, yet the diversity of over 7,000 sign languages worldwide presents significant challenges for automated recognition systems. This paper presents a novel approach to sign language recognition (SLR) that integrates computer vision techniques with advanced natural language processing (NLP) to improve transcription accuracy and contextual relevance. Our system employs a two-stage architecture: first, a gesture recognition component utilizing MediaPipe Holistic for landmark extraction and Long Short-Term Memory (LSTM) networks for classification; second, a text enhancement module using bidirectional LSTM for contextual correction and grammatical improvement. Experimental results demonstrate that our NLP-enhanced system achieves 98.46% accuracy in gesture recognition while significantly improving the grammatical correctness and contextual coherence of the generated text compared to systems without NLP enhancement. The system can successfully identify missing function words, add appropriate punctuation, and correct grammatical errors in real-time. While primarily focused on American Sign Language (ASL), our approach provides valuable insights for developing more effective and inclusive SLR technologies for various sign languages. These advancements represent a meaningful step toward bridging communication gaps between signing and non-signing individuals, potentially enhancing accessibility in educational, professional, and social environments.

• sign language recognition
• American Sign Language
• gesture recognition
• MediaPipe Holistic
• landmark extraction
• LSTM
• bidirectional LSTM
• text enhancement
• grammatical error correction
• punctuation correction
• sign language transcription
• deaf and hard-of-hearing communication
• accessibility

• Hand Gesture Recognition Systems
• Hearing Impairment and Communication
• Human Pose and Action Recognition
• sign language recognition
• American Sign Language
• gesture recognition
• MediaPipe Holistic
• landmark extraction
• LSTM
• bidirectional LSTM
• text enhancement
• Human-Computer Interaction
• Developmental and Educational Psychology
• Computer Vision and Pattern Recognition
• Computer Science
• Psychology