Sri Lankan Sign Language to Sinhala Text using Convolutional Neural Network Combined with Scale Invariant Feature Transform (SIFT)

Abstract

Sri Lankan sign language (SSL) is a visualgestural language used by the deaf community for communication. Hearing-impaired people cannot effectively communicate with a normal person due to the difficulty in understanding sign language. SSL to Sinhala text interpreting technology using gesture recognition helps to fill up this communication gap since Sinhala is the majority language used in Sri Lanka. Hand gesture recognition can be achieved by using vision-based or sensor-based approaches. Vision-based approaches are comparatively simple and less costly but sensorbased approaches are complex. Scale, rotation, occlusion affects the accuracy of gesture recognition, and keypoints act as better features to handle them. The research focuses on a combined approach of convolutional neural network (CNN) and Scale Invariant Feature Transform (SIFT) to develop a camera-based low-cost solution to interpret static gestures of SSL into Sinhala text. The SSL to Sinhala text translation model reached an accuracy of 86.5% when a dataset of images of 20 static SSL gestures was used. The classifier showed robustness to scale variations when the distance to the camera was varied and uniform color backgrounds were used. Further improvements can be done for the recognition of dynamic gestures and facial expressions of SSL.