Assessing Distribution Shift in Human Activity Reco...

Assessing Distribution Shift in Human Activity Reco... | AI Research

Key Takeaways

Assessing Distribution Shift in Human Activity Recognition for Domain Generalization Human Activity Recognition (HAR) systems often struggle when moved from...
While the field of Human Activity Recognition (HAR) continues to draw interest from researchers and advance in important ways, some key challenges remain.
Towards that end, this paper systematically evaluates 4 different types of distribution shifts, including variations in device type, sensor placement, sampling rate, and user behavior.
Quantifying their effects, we illustrate that diversity shifts predominantly define all types of shifts, indicating the existence of unique features that are not shared across different domains.
We then introduce a uniform HAR-based distribution shift benchmarks and conduct a comprehensive evaluation of up to 28 domain generalization methods.

Paper AbstractExpand

While the field of Human Activity Recognition (HAR) continues to draw interest from researchers and advance in important ways, some key challenges remain. One of the most difficult aspects of building HAR models that show good performance in real-world settings is dealing with data diversity from device and sensor heterogeneity, and contextual changes that are intrinsic to real-world applications. While data diversity in HAR has been well-acknowledged in the literature, there remains a gap in understanding the effect of various types of distribution shifts on HAR models and the domain generalization problem that arises. Towards that end, this paper systematically evaluates 4 different types of distribution shifts, including variations in device type, sensor placement, sampling rate, and user behavior. Quantifying their effects, we illustrate that diversity shifts predominantly define all types of shifts, indicating the existence of unique features that are not shared across different domains. We then introduce a uniform HAR-based distribution shift benchmarks and conduct a comprehensive evaluation of up to 28 domain generalization methods. Our analysis exposes the limitations of current domain generalization algorithms in achieving model generalizability, marginally outperforming the empirical risk minimization baseline. This work represents the first systematic exploration of domain generalization and adaptation concerning specific distribution shifts in sensor-based HAR, offering an open-source benchmark platform and datasets to spur further research.

Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
Human Activity Recognition (HAR) systems often struggle when moved from controlled laboratory settings to the real world. This performance drop occurs because models trained on one set of conditions—such as a specific device or sensor placement—frequently fail to adapt when those conditions change. This paper investigates why these "distribution shifts" occur and evaluates how well current machine learning algorithms handle the challenge of generalizing to new, unseen environments without needing additional training data.

Understanding Distribution Shifts

The researchers identified four primary factors that cause data distributions to shift in HAR applications: device type, sensor placement, sampling rate, and user behavior. To understand these shifts, the authors quantified them using a two-dimensional spectrum that measures both "diversity shift" and "correlation shift." Their analysis revealed that diversity shifts are the dominant factor across all categories. This suggests that different domains contain unique, non-shared features, which makes it difficult for a model trained in one environment to perform reliably in another.

Building a New Benchmark

To test how well current models handle these challenges, the authors created a uniform benchmark platform. They utilized several existing public datasets to represent shifts in sensor location, sampling rate, and device type. Additionally, they conducted a new study focused on user behavior, where they tracked participants as they learned to juggle. By recording accelerometer data as participants progressed from juggling three balls to seven, the researchers were able to capture how changes in skill level and movement patterns create distinct distribution shifts.

Evaluating Model Performance

Using this new benchmark, the team evaluated 28 different domain generalization algorithms. The results were revealing: despite the variety of sophisticated methods tested, most struggled to significantly outperform a basic approach known as empirical risk minimization. This finding highlights a major gap in current technology, demonstrating that existing domain generalization algorithms are not yet robust enough to handle the complexities of sensor-based heterogeneity.

Key Takeaways for Future Research

This work serves as the first systematic exploration of domain generalization specifically tailored to sensor-based HAR. By providing an open-source benchmark platform and datasets, the authors aim to shift the focus of the field toward addressing these specific, real-world hurdles. The study emphasizes that until models can better account for the unique features introduced by different devices and user behaviors, they will remain limited in their ability to function effectively in dynamic, everyday environments.