Dual-Attention ResNet50-LSTM for Video-Based Human Action Recognition

Abstract—Human action recognition (HAR) from video requires effective modeling of both spatial appearance and temporal dynamics. Although convolutional neural networks (CNNs) can extract discriminative spatial features from individual frames, they are limited in modeling long-term temporal dependencies. Long Short-Term Memory (LSTM) networks address this limitation by learning sequential representations; however,
conventional CNN-LSTM frameworks may assign equal importance to irrelevant spatial regions and temporal observations. This paper proposes a dual-attention ResNet50-LSTM framework for video-based human action recognition. A pretrained ResNet50 network first extracts spatial feature maps from uniformly sampled video frames. A spatial attention mechanism then identifies informative regions within each frame, while an
attention gate regulates the attended features before sequential processing by an LSTMCell. Subsequently, a temporal attention mechanism assigns adaptive importance weights to hidden representations across the video sequence for final classification. The proposed framework is evaluated on the UCF50 and UCF101 benchmark datasets and achieves test accuracies of 94.10% and 92.42%, respectively. The results demonstrate the effectiveness of combining spatial and temporal attention with CNN-LSTM based
video representation learning.

The major contributions of this work are summarized as follows:
• A dual-attention ResNet50-LSTM framework is proposed for video-based human action recognition.
• A spatial attention mechanism is employed to emphasize discriminative regions within individual video frames.
• An attention-gating mechanism regulates the attended spatial features before sequential processing.
• A temporal attention mechanism is used to adaptively aggregate LSTM hidden representations.
• The framework is evaluated on UCF50 and UCF101, achieving test accuracies of 94.10% and 92.42%, respectively.