Cross-Sign Language Transfer Learning Using Domain... | AI Research

Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment addresses the scarcity of recognition resources for the world's over 100 distinct sign languages. Researchers Keren Artiaga, Yang Li, Ercan Engin Kuruoglu, and Wai Kin Chan propose a method to improve Sign Language Recognition (SLR) by transferring knowledge from high-resource sign language datasets to low-resource ones, specifically using American Sign Language (ASL) as the target domain.

Addressing Data Scarcity

Because most sign languages lack large-scale datasets, models often suffer from overfitting. While common approaches rely on pre-training models on general datasets like ImageNet, the authors argue that this provides only superficial features. Instead, they utilize domain adaptation to transfer knowledge directly between sign languages—specifically using Argentine Sign Language (LSA) and Chinese Sign Language (CSL) as source domains to improve ASL recognition. This approach aims to capture high-level, language-specific visual features that general datasets miss.

The Role of Multi-scale Temporal Alignment

The study employs the Temporal Attentive Adversarial Adaptation Network (TA3N), which incorporates a Temporal Relation Network (TRN) module. This module aligns temporal relations—sequences of time-ordered frames—between the source and target domains. By assigning weights to these relations, the model focuses on the most domain-discriminative features. The researchers tested five different multi-scale temporal relations (3, 5, 7, 10, and 15) to determine the optimal timescale for alignment.

Key Findings

The researchers found that aligning shorter-term temporal relations (3, 5, and 7) is more effective than aligning longer-term ones (10 and 15) for improving the accuracy of the target SLR model. This held true across both LSA-to-ASL and CSL-to-ASL adaptations. Additionally, the study compared RGB and Optical Flow input modalities, concluding that RGB frames outperformed Optical Flow in the majority of cases. The team also demonstrated that their method functions in both full-scale and few-shot transfer learning settings, the latter of which requires significantly less training data.

Limitations and Considerations

The authors note that their work is constrained by the quality of existing datasets. For instance, the ASL samples used in the study were sourced from various online platforms, resulting in inconsistent lighting, backgrounds, and subject distances, as well as lower video quality compared to the LSA and CSL datasets. Furthermore, the study focuses on isolated sign language gestures, and the researchers emphasize that their methodology is designed to be universally applicable to any sign language recognition task, despite the specific languages tested.