Open Access

ARTICLE

Prediction of Assembly Intent for Human-Robot Collaboration Based on Video Analytics and Hidden Markov Model

Jing Qu¹, Yanmei Li^1,2, Changrong Liu¹, Wen Wang¹, Weiping Fu^1,3,*

1 School of Mechanical and Precision Instrument Engineering, Xi’an University of Technology, Xi’an, 710048, China
2 Liupanshan Laboratory, Yinchuan, 750021, China
3 School of Engineering, Xi’an International University, Xi’an, 710077, China

* Corresponding Author: Weiping Fu. Email:

(This article belongs to the Special Issue: Applications of Artificial Intelligence in Smart Manufacturing)

Computers, Materials & Continua 2025, 84(2), 3787-3810. https://doi.org/10.32604/cmc.2025.065895

Received 24 March 2025; Accepted 19 May 2025; Issue published 03 July 2025

Abstract

Despite the gradual transformation of traditional manufacturing by the Human-Robot Collaboration Assembly (HRCA), challenges remain in the robot’s ability to understand and predict human assembly intentions. This study aims to enhance the robot’s comprehension and prediction capabilities of operator assembly intentions by capturing and analyzing operator behavior and movements. We propose a video feature extraction method based on the Temporal Shift Module Network (TSM-ResNet50) to extract spatiotemporal features from assembly videos and differentiate various assembly actions using feature differences between video frames. Furthermore, we construct an action recognition and segmentation model based on the Refined-Multi-Scale Temporal Convolutional Network (Refined-MS-TCN) to identify assembly action intervals and accurately acquire action categories. Experiments on our self-built reducer assembly action dataset demonstrate that our network can classify assembly actions frame by frame, achieving an accuracy rate of 83%. Additionally, we develop a Hidden Markov Model (HMM) integrated with assembly task constraints to predict operator assembly intentions based on the probability transition matrix and assembly task constraints. The experimental results show that our method for predicting operator assembly intentions can achieve an accuracy of 90.6%, which is a 13.3% improvement over the HMM without task constraints.

---

Keywords

---