Monocular 3D Human Pose Estimation-Based Joystick Control System for Teleoperation of a Cobot Arm

Teleoperation is the process of remotely controlling a robotic system. This paper compares two real-time 3D human pose estimation (HPE) models for teleoperating the tool centre point (TCP) of a UR5 robot arm on non-GPU hardware: BlazePose and a lightweight version of OpenPose. These models were selected for their real-time capabilities. Building on previous work, the 3D joint position of the right hand, derived from monocular RGB input, is used to generate remote control instructions for the UR5, while the left hand controls the attached RG2 gripper. A new joystick-based control system is introduced, where right-hand movement in any direction translates to continuous robot TCP movement in that direction at a fixed speed. This addresses issues with the prior mapping-based system, which relied on accurate depth estimation and often caused unintended arm movements. The two HPE models were evaluated on hold-position and pick-and-place tasks, comparing both the joystick and mapping control systems. Results indicate the lightweight OpenPose model reduces unintended movement but remains insufficient for pick-and-place tasks under the mapping system. In contrast, the joystick system enabled successful task completion with accuracy comparable to previous 2D control methods, while freeing the left hand for additional controls. Future work may enhance the system by using the left hand for rotational control.