 ICACT20260043 Slide.01
[Big Slide]
|
Chrome  Click!! |
 |
Hi everyone, and welcome to our presentation. My name is Devesh Singh, and along with my colleague Dr. Manoj Kumar Gupta, we represent the School of Computer Science and Engineering at Shri Mata Vaishno Devi University.
Today, we are presenting our research titled "Real-Time Indian Sign Language Interpretation by Deep Neural Network and MediaPipe Framework". We are honored to share this work at the 28th International Conference on Advanced Communications Technology (ICACT 2026). Our focus today is on how we can leverage ubiquitous AI to solve critical accessibility challenges. |
| ICACT20260043 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
We wanted to align our work directly with this years special slogan: "Exploring the Ubiquitous Artificial Intelligence". When we say "ubiquitous," we mean AI that is present everywhere and accessible to everyone.
Our core contribution addresses this by shifting sign language interpretation away from high-performance computing clusters and moving it onto the standard, low-resource consumer devices that people already own. We believe true accessibility means technology must run on laptops and mobile phones without requiring expensive GPUs or specialized sensors. To achieve this, we executed a technological shift from heavy Convolutional Neural Networks (CNNs) to lightweight Feedforward Neural Networks (FFNNs). Our goal is simple: to make ISL interpretation as ubiquitous as the devices we carry. |
| ICACT20260043 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
Let’s look at the Global Context to understand the scale of the need. According to World Health Organization (WHO) data, it is estimated that approximately 430 million individuals worldwide suffer from disabling hearing loss.
In a world dominated by spoken languages, these individuals face a massive communication barrier, leading to exclusion from education, healthcare, and employment. While Sign Language functions as their primary mode of communication complete with its own grammar and syntax the vast majority of the hearing population simply does not understand it, creating a persistent wall between these two groups. |
| ICACT20260043 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Bringing this context to India, the crisis is even more acute. India has a massive deaf population estimated at roughly 70 million individuals, which is about 6.3% of our population. Approximately 18 million people have severe hearing impairments and are dependent on sign language although only a subset uses standardized ISL daily.
However, we face a critical Interpreter Shortage. There are only an estimated 250 to 350 certified ISL interpreters in the entire country. This results in a staggering ratio imbalance where hundreds of thousands of users rely on a single interpreter, making human interpretation impossible for daily interactions. |
| ICACT20260043 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
This leads us to our Problem Statement. First, the severe shortage of human interpreters creates a dependency where deaf individuals lack support in critical situations like hospitals or banks.
Second, there is a Technological Gap; existing tools are either too expensive or require complex hardware, making them unviable for consumers. Third, there is a Real-Time Requirement communication happens instantly, so offline rendering is useless. Finally, ISL is complex, involving two-handed movements and rapid transitions that are hard to track. Our challenge was to develop a system that is accurate, fast, AND runs on a standard laptop. |
| ICACT20260043 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
We analyzed existing solutions, starting with Glove-Based Systems. These use flex sensors and accelerometers to capture motion directly.
While they offer high accuracy regarding joint angles, they have three major limitations. First, they are invasive and cumbersome to wear. Second, the hardware is expensive and fragile. And third, issues with hygiene and maintenance add friction to daily usage. |
| ICACT20260043 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Next, we looked at Vision-Based Systems. These use cameras to track hands, which is a major advantage because it is non-invasive.
However, traditional approaches relied on skin color segmentation, which fails if the lighting changes or background skin tones match the user. Newer Deep Learning approaches using 3D CNNs are accurate but computationally heavy. Many also require specialized hardware like depth cameras (Kinect/RealSense), which limits portability. |
| ICACT20260043 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
Our Literature Review highlighted several key studies. Das et al. used Inception V3 CNNs but required heavy image processing and cropping. Sahoo found k-NN effective but limited the scope to just digits.
Ansari & Harit achieved high accuracy but were tied to the non-portable Kinect sensor. Finally, Bora et al. used MediaPipe, which was a direct precursor to our work, but they were limited to only 9 single-handed gestures. |
| ICACT20260043 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
This review helped us identify specific Research Gaps. Previous studies lacked the combination of features needed for a truly "Ubiquitous" tool.
First, there was a Limited Vocabulary Scope; many lightweight models ignored the complex two-handed nature of full ISL. Second, Hardware Constraints meant that high-accuracy CNNs often ran at less than 10 FPS on standard CPUs. And third, most datasets focused on perfectly cropped images rather than realistic webcam inputs. |
| ICACT20260043 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
To address these gaps, we set four clear Research Objectives:
1. Develop a Real-Time Interpreter capable of processing 30+ FPS on a standard laptop CPU.
2. Expand Gesture Support to 36 classes (A-Z, 0-9), covering both one-handed and two-handed static gestures.
3. Ensure Privacy by designing an architecture that processes only anonymized coordinate points, never storing video.
4. Maximize Accuracy to over 98% using a lightweight architecture suitable for Edge AI. |
| ICACT20260043 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
Here is a high-level view of our System Pipeline. It is designed for minimal latency across three stages.
Stage 1 handles Input Acquisition and Hand Detection. Stage 2 focuses on Landmark Extraction, Data Padding, and Normalization. Finally, Stage 3 is where our Feedforward Neural Network performs the training and real-time prediction. |
| ICACT20260043 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
Lets break that pipeline down.
In Stage 1, we capture live video using OpenCV and use the MediaPipe framework to detect hands and extract 21 landmarks per hand. In Stage 2, we normalize these raw coordinates to a -1 to 1 range and pad them for consistency. In Stage 3, our custom FFNN predicts the sign class from the processed vector, and the output is overlaid on the video feed in real-time. |
| ICACT20260043 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
A key question is: Why MediaPipe?. MediaPipe is a cross-platform framework by Google that is specifically optimized for mobile and edge devices.
Unlike raw ResNet or VGG models, MediaPipe is lightweight and robust—it successfully tracks hands even with self-occlusions and varying lighting. By handling the heavy computer vision tasks, it allows our custom model to focus purely on the coordinate math, saving massive computational resources. |
| ICACT20260043 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
For Feature Extraction, MediaPipe gives us 21 distinct fiducial points for each detected hand, such as the Wrist and Finger Tips.
Each point has (x, y, z) coordinates, but we explicitly discard the z-coordinate to reduce noise, as webcam depth estimation is unreliable. This leaves us with a feature vector of 42 features per hand (21 points × 2 axes). For two hands, our total input is 84 features. |
| ICACT20260043 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
Raw coordinates can be problematic because they change if a user stands closer or further away. To solve this, we applied Normalization Logic.
First, we ensure Translation Invariance by setting the wrist as the origin (0,0) for every frame. Second, we ensure Scale Invariance by resizing all values to fall between -1 and 1. The result is that a gesture looks mathematically identical whether performed 1 meter or 3 meters from the camera. |
| ICACT20260043 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
One of our biggest technical challenges was handling One vs. Two Hands. ISL contains both, but neural networks require a fixed input size.
Our strategy was to define a fixed input vector size of 84. If two hands are detected, we fill the vector completely. If only one hand is detected, we fill the first 42 slots and pad the remaining 42 slots with zeros. This allows a single model to seamlessly switch between one and two-handed signs without crashing. |
| ICACT20260043 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
For Data Collection, we created an Author-Generated Dataset to ensure strict ethical compliance and privacy.
We collected data in a semi-controlled environment using a standard 720p webcam, introducing variability in lighting, distance, and angles. Crucially, our privacy protocol ensured that no raw images were saved; only the anonymized (x,y) landmark vectors were logged to our CSV files. |
| ICACT20260043 Slide.18
[Big Slide]
|
Chrome Click!! |
 |
Here are our Dataset Statistics. We collected a total volume of 36,200 data points covering 36 classes (26 Alphabets and 10 Digits).
As you can see from the chart, we maintained a balanced distribution of approximately 1,000 samples per class. This balance is critical to prevent the model from developing a bias toward more frequent signs. |
| ICACT20260043 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
For our model, we implemented a straightforward Feedforward Neural Network (FFNN) using Keras.
It consists of an Input Layer of 84 neurons, followed by two Hidden Layers with 64 and 32 neurons respectively, both using ReLU activation. We used Dropout layers to randomly disable neurons and prevent overfitting. The final Output Layer has 36 neurons with Softmax activation to provide the class probability. |
| ICACT20260043 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
The beauty of this architecture lies in its Model Complexity & Size. The model has only 8,708 trainable parameters, resulting in a physical file size of just 34 KB.
To put that in perspective, standard CNNs like VGG16 are often 100MB to 500MB. A 34 KB model implies true Ubiquity it can literally run on a microcontroller, a cheap IoT chip, or a legacy smartphone with almost zero memory footprint. |
| ICACT20260043 Slide.21
[Big Slide]
|
Chrome Click!! |
 |
For Training Configuration, we used the Adam optimizer for its efficiency and Sparse Categorical Cross-Entropy as our loss function.
We set the max epochs to 1000 but utilized Early Stopping. As the graph shows, the training stopped automatically at Epoch 199 when the validation loss stabilized, preventing wasted computation. |
| ICACT20260043 Slide.22
[Big Slide]
|
Chrome Click!! |
 |
Lets look at the Results: Classification Metrics. Evaluating on our sample test set, we achieved an Overall Accuracy of 99%.
Our Precision, Recall, and F1-Scores all average around 0.99. High performance was observed across both simple single-handed digits and the more complex alphabet signs, validating our padding strategy. |
| ICACT20260043 Slide.23
[Big Slide]
|
Chrome Click!! |
 |
This Class-wise Precision chart visually confirms our results. You can see the bars are consistently high across all 36 classes, indicating a very low false positive rate for every single gesture. |
| ICACT20260043 Slide.24
[Big Slide]
|
Chrome Click!! |
 |
Similarly, the Class-wise Recall chart shows that the model rarely misses a gesture. The consistency here is key it means the system is reliable regardless of which sign the user is performing. |
| ICACT20260043 Slide.25
[Big Slide]
|
Chrome Click!! |
 |
The F1-Score chart combines these metrics to show the strong balance between precision and recall. Achieving nearly 1.0 across the board confirms the robustness of our lightweight architecture. |
| ICACT20260043 Slide.26
[Big Slide]
|
Chrome Click!! |
 |
Finally, the Class-wise Support chart simply confirms that our testing methodology was sound. We tested on a balanced number of samples for each class, ensuring that our high accuracy metrics are statistically significant. |
| ICACT20260043 Slide.27
[Big Slide]
|
Chrome Click!! |
 |
Now, lets talk about Real-Time Performance. We tested this on an AMD Ryzen 7 laptop with no dedicated GPU.
We achieved an average inference speed of 30 FPS, which translates to a latency of about 33ms per frame. This is crucial because real-time feedback is necessary for conversation; anything slower than 15 FPS feels "laggy" and breaks the flow. We have successfully achieved the "Real-Time" benchmark on consumer hardware. |
| ICACT20260043 Slide.28
[Big Slide]
|
Chrome Click!! |
 |
We achieved this speed using specific Optimization Techniques.
First, we used Asynchronous Multi-threading to decouple the webcam frame capture from the model inference, preventing the display from freezing. Second, we implemented a Stationarity Check if the hand moves less than a certain pixel threshold, we skip the inference and reuse the previous result. This reduces computational load and prevents "flickering" predictions when the hand is held still.
|
| ICACT20260043 Slide.29
[Big Slide]
|
Chrome Click!! |
 |
Comparing with the State-of-the-Art, the work by Bora et al. (2023) was our closest benchmark.
While they also used MediaPipe and FFNN, they were limited to 9 single-handed classes. Our work scaled this up to 36 classes a 4x increase and added support for Two-handed gestures, all while maintaining the same 99% accuracy. This proves that our architecture scales effectively to larger vocabularies. |
| ICACT20260043 Slide.30
[Big Slide]
|
Chrome Click!! |
 |
When we Compare vs. CNN Approaches, the difference is even more stark.
Traditional CNNs process over 150,000 inputs (pixels) per frame. Our approach processes only 84 inputs that is approximately 0.05% of the input size of a CNN. This explains our massive speed advantage and small 34 KB model size, while still matching the accuracy of heavy 3D CNNs. |
| ICACT20260043 Slide.31
[Big Slide]
|
Chrome Click!! |
 |
We also prioritized Privacy & Ethical Compliance. By design, our architecture processes video streams in RAM only no video footage or images are ever written to the disk.
By strictly practicing Data Minimization and processing only 42 pairs of numbers, it is impossible to reconstruct a users face or identity. This ensures we align with GDPR and Indias new DPDP Act 2023. |
| ICACT20260043 Slide.32
[Big Slide]
|
Chrome Click!! |
 |
In Conclusion, we successfully developed a real-time, lightweight, and private ISL interpreter that runs on standard hardware, fulfilling the vision of Ubiquitous AI.
While our current limitation is that we support only static gestures , our Future Work involves integrating LSTMs to recognize dynamic, time-series gestures and incorporating facial landmarks to capture grammatical nuances. This work is a foundational step toward inclusive, low-cost assistive technology for the 70 million hearing-impaired individuals in India. |
| ICACT20260043 Slide.33
[Big Slide]
|
Chrome Click!! |
 |
Here is a list of the References we cited throughout this presentation, covering previous work in glove-based systems, vision systems, and sign language recognition. |
| ICACT20260043 Slide.34
[Big Slide]
|
Chrome Click!! |
 |
Here is a Continued list of the References we cited throughout this presentation, covering previous work in glove-based systems, vision systems, and sign language recognition. |
| ICACT20260043 Slide.35
[Big Slide]
|
Chrome Click!! |
 |
Thank you very much for your time and attention. We are now happy to take any questions you may have. |
